Establishing a video conference during a phone call

ABSTRACT

Some embodiments provide a method for initiating a video conference using a first mobile device. The method presents, during an audio call through a wireless communication network with a second device, a selectable user-interface (UI) item on the first mobile device for switching from the audio call to the video conference. The method receives a selection of the selectable UI item. The method initiates the video conference without terminating the audio call. The method terminates the audio call before allowing the first and second devices to present audio and video data exchanged through the video conference.

BACKGROUND

Many of today's portable devices, such as smartphones, provide videocapture functionality. A user of the portable device can capture bothstill images and video through a camera on the phone. However, totransmit captured video to another party, the user must generally eithersend the video directly to the other party or upload the video toanother location (e.g., an Internet video hosting site) after the videois done being captured. Unfortunately, this does not allow the otherparty to view the live video stream as it is captured by the portabledevice.

In addition, standard portable devices are only equipped with onecamera, and processing information from this one camera is difficultenough. An ideal device would have multiple cameras and could send outlive video that is a composition of video from at least two cameras.This is an especially difficult problem in light of the limitedresources available for portable devices, both in terms of the deviceprocessing multiple captured video streams and a network to which thedevice is connected handling the transmission of the live video streams.

BRIEF SUMMARY

Some embodiments of the invention provide a mobile device with twocameras that can take pictures and videos. The mobile device of someembodiments has a display screen for displaying the captured pictureimages and video images. It also includes a storage for storing thecaptured images for later transmission to another device. The devicefurther has a network interface that allows the device to transmit thecaptured images to one or more devices during a real-time communicationsession between the users of the devices. The device also includes anencoder that it can use to encode the captured images for local storageor for transmission to another device. The mobile device furtherincludes a decoder that allows the device to decode images captured byanother device during a real-time communication session or to decodeimages stored locally.

One example of a real-time communication session that involves thetransmission of the captured video images is a video conference. In someembodiments, the mobile device can only transmit one camera's capturedvideo images at any given time during a video conference. In otherembodiments, however, the mobile device can transmit captured videoimages from both of its cameras simultaneously during a video conferenceor other real-time communication session.

During a video conference with another device, the mobile device of someembodiments can transmit other types of content along with the videocaptured by one or both of its cameras. One example of such othercontent includes low or high resolution picture images that are capturedby one of the device's cameras, while the device's other camera iscapturing a video that is used in the video conference. Other examplesof such other content include (1) files and other content stored on thedevice, (2) the screen display of the device (i.e., the content that isdisplayed on the device's screen), (3) content received from anotherdevice during a video conference or other real-time communicationsession, etc.

The mobile devices of some embodiments employ novel in-conferenceadjustment techniques for making adjustments during a video conference.For instance, while transmitting only one camera's captured video duringa video conference, the mobile device of some embodiments candynamically switch to transmitting a video captured by its other camera.In such situations, the mobile device of some embodiments notifies anyother device participating in the video conference of this switch sothat this other device can provide a smooth transition on its endbetween the videos captured by the two cameras.

In some embodiments, the request to switch cameras not only canoriginate on the “local” device that switches between its cameras duringthe video conference, but also can originate from the other “remote”device that is receiving the video captured by the local device.Moreover, allowing one device to direct another device to switch camerasis just one example of a remote control capability of the devices ofsome embodiments. Examples of other operations that can be directed to adevice remotely in some embodiments include exposure adjustmentoperations (e.g., auto-exposure), focus adjustment operations (e.g.,auto-focus), etc. Another example of a novel in-conference adjustmentthat can be specified locally or remotely is the identification of aregion of interest (ROI) in a captured video, and the use of this ROIidentification to modify the behavior of the capturing camera, to modifythe image processing operation of the device with the capturing camera,or to modify the encoding operation of the device with the capturingcamera.

Yet another example of a novel in-conference adjustment of someembodiments involves real-time modifications of composite video displaysthat are generated by the devices. Specifically, in some embodiments,the mobile devices generate composite displays that simultaneouslydisplay multiple videos captured by multiple cameras of one or moredevices. In some cases, the composite displays place the videos inadjacent display areas (e.g., in adjacent windows). In other cases, thecomposite display is a picture-in-picture (PIP) display that includes atleast two display areas that show two different videos where one of thedisplay areas is a background main display area and the other is aforeground inset display area that overlaps the background main displayarea.

The real-time modifications of the composite video displays in someembodiments involve moving one or more of the display areas within acomposite display in response to a user's selection and movement of thedisplay areas. Some embodiments also rotate the composite display duringa video conference, when the screen of the device that provides thiscomposite display rotates. Also, the mobile device of some embodimentsallows the user of the device to swap the videos in a PIP display (i.e.,to make the video in the foreground inset display appear in thebackground main display while making the video in the background maindisplay appear in the foreground inset display).

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates a composite display of some embodiments.

FIG. 2 illustrates another composite display of some embodiments.

FIG. 3 conceptually illustrates a software architecture for a videoprocessing and encoding module of a dual camera mobile device of someembodiments.

FIG. 4 conceptually illustrates a captured image processing unit of someembodiments.

FIG. 5 conceptually illustrates examples of different frame rates basedon different vertical blanking intervals (VBIs).

FIG. 6 conceptually illustrates examples of different interleaving framerates based on different VBIs.

FIG. 7 conceptually illustrates another captured image processing unitof some embodiments.

FIG. 8 conceptually illustrates another captured image processing unitof some embodiments.

FIG. 9 conceptually illustrates a software architecture for a videoconferencing and processing module of a dual camera mobile device ofsome embodiments.

FIG. 10 conceptually illustrates an example video conference requestmessaging sequence of some embodiments.

FIG. 11 illustrates a user interface of some embodiments for a videoconference setup operation.

FIG. 12 illustrates a user interface of some embodiments for acceptingan invitation to a video conference.

FIG. 13 illustrates another user interface of some embodiments foraccepting an invitation to a video conference.

FIG. 14 illustrates another user interface of some embodiments for avideo conference setup operation.

FIG. 15 conceptually illustrates a process of some embodiments forsetting a bit rate for a video conference.

FIG. 16 conceptually illustrates another software architecture for avideo conferencing and processing module of a dual camera mobile deviceof some embodiments.

FIG. 17 conceptually illustrates another software architecture for adual camera mobile device of some embodiments.

FIG. 18 conceptually illustrates a process performed by a videoconference manager of some embodiments such as that illustrated in FIG.16 .

FIG. 19 conceptually illustrates a software architecture for a temporalnoise reduction module of some embodiments.

FIG. 20 conceptually illustrates a process of some embodiments forreducing temporal noise of images of a video.

FIG. 21 conceptually illustrates a process performed by an imageprocessing manager of some embodiments such as that illustrated in FIG.9 .

FIG. 22 illustrates a user interface of some embodiments for an exposureadjustment operation.

FIG. 23 illustrates a user interface of some embodiments for a focusadjustment operation.

FIG. 24 conceptually illustrates a perspective correction processperformed by an image processing manager of some embodiments such asthat illustrated in FIG. 16 .

FIG. 25 conceptually illustrates example perspective correctionoperations of some embodiments.

FIG. 26 conceptually illustrates a software architecture for an encoderdriver of some embodiments of some embodiments such as that illustratedin FIG. 16 .

FIG. 27 conceptually illustrates an image resizing process performed byan encoder driver of some embodiments such as that illustrated in FIG.26 .

FIG. 28 conceptually illustrates a software architecture for a decoderdriver of some embodiments such as that illustrated in FIG. 16 .

FIG. 29 conceptually illustrates an image extraction process performedby a decoder driver of some embodiments such as that illustrated in FIG.28 .

FIG. 30 illustrates an encoder driver of some embodiments that includestwo rate controllers.

FIG. 31 conceptually illustrates a software architecture for anetworking manager of some embodiments such as that illustrated in FIG.16 .

FIG. 32 illustrates a user interface of some embodiments for asnap-to-corner operation.

FIG. 33 illustrates another user interface of some embodiments for asnap-to-corner operation.

FIG. 34 illustrates a user interface of some embodiments for a PIPdisplay rotation operation.

FIG. 35 illustrates another user interface of some embodiments for a PIPdisplay rotation operation.

FIG. 36 illustrates another user interface of some embodiments for a PIPdisplay rotation operation.

FIG. 37 illustrates another user interface of some embodiments for a PIPdisplay rotation operation.

FIG. 38 illustrates a user interface of some embodiments for resizing aforeground inset display area in a PIP display.

FIG. 39 illustrates another user interface of some embodiments forresizing an inset display area in a PIP display.

FIG. 40 illustrates another user interface of some embodiments forresizing an inset display area in a PIP display.

FIG. 41 illustrates another user interface of some embodiments forresizing an inset display area in a PIP display.

FIG. 42 illustrates a user interface of some embodiments for identifyinga region of interest in a display.

FIG. 43 illustrates another user interface of some embodiments foridentifying a region of interest in a display.

FIG. 44 illustrates another user interface of some embodiments foridentifying a region of interest in a display.

FIG. 45 illustrates a process of some embodiments for performing a localswitch camera operation on a dual camera mobile device.

FIG. 46 illustrates a user interface of some embodiments for a switchcamera operation.

FIG. 47 illustrates another user interface of some embodiments for aswitch camera operation.

FIG. 48 illustrates another user interface of some embodiments for aswitch camera operation.

FIG. 49 illustrates another user interface of some embodiments for aswitch camera operation.

FIG. 50 illustrates a process of some embodiments for performing aremote switch camera operation on a dual camera mobile device.

FIG. 51 illustrates a user interface of some embodiments for a remotecontrol switch camera operation.

FIG. 52 illustrates another user interface of some embodiments for aremote control switch camera operation.

FIG. 53 illustrates another user interface of some embodiments for aremote control switch camera operation.

FIG. 54 illustrates another user interface of some embodiments for aremote control switch camera operation.

FIG. 55 conceptually illustrates a process of some embodiments forperforming an exposure adjustment operation.

FIG. 56 illustrates a user interface of some embodiments for an exposureadjustment operation.

FIG. 57 illustrates another user interface of some embodiments for anexposure adjustment operation.

FIG. 58 illustrates another user interface of some embodiments for anexposure adjustment operation.

FIG. 59 conceptually illustrates an exposure adjustment processperformed by an image processing manager of some embodiments such asthat illustrated in FIG. 16 .

FIG. 60 conceptually illustrates exposure adjustment operations of someembodiments.

FIG. 61 conceptually illustrates a process of some embodiments forperforming a focus adjustment operation.

FIG. 62 illustrates a user interface of some embodiments for a focusadjustment operation.

FIG. 63 illustrates another user interface of some embodiments for afocus adjustment operation.

FIG. 64 illustrates another user interface of some embodiments for afocus adjustment operation.

FIG. 65 illustrates different display arrangements of some embodimentsfor videos captured from one or more dual camera mobile devices.

FIG. 66 illustrates a user interface of some embodiments forsuperimposing a foreground of an inset video onto a background video ina PIP display.

FIG. 67 illustrates a technique of some embodiments for determining aforeground of video images.

FIG. 68 illustrates a user interface of some embodiments for swapping aninset display with a background display in a PIP display during a videoconference.

FIG. 69 illustrates a user interface of some embodiments for asnap-to-corner operation.

FIG. 70 illustrates a user interface of some embodiments for asnap-to-corner and push operation.

FIG. 71 illustrates a user interface of some embodiments for a PIPdisplay rotation operation.

FIG. 72 illustrates another user interface of some embodiments for a PIPdisplay rotation operation.

FIG. 73 illustrates a user interface of some embodiments for selectingone video from two remote videos during a video conference.

FIG. 74 illustrates a user interface of some embodiments for selectingone video from two local videos during a video conference.

FIG. 75 illustrates a user interface of some embodiments for apre-conference selection of a video to use for the video conference.

FIG. 76 illustrates examples of bandwidth allocation between two videoscaptured by a dual camera mobile device of some embodiments.

FIG. 77 conceptually illustrates an arbitrator module of someembodiments for managing rate controllers of a dual camera mobiledevice.

FIG. 78 conceptually illustrates a method of some embodiments forencoding images captured by cameras of a dual camera mobile device.

FIG. 79 conceptually illustrates another method of some embodiments forencoding images captured by cameras of a dual camera mobile device.

FIG. 80 illustrates example image composites for the method illustratedin FIG. 79 .

FIG. 81 conceptually illustrates another method of some embodiments forencoding images captured by cameras of a dual camera mobile device.

FIG. 82 conceptually illustrates a method of some embodiments fordecoding images captured by cameras of a dual camera mobile device.

FIG. 83 conceptually illustrates another method of some embodiments fordecoding images captured by cameras of a dual camera mobile device.

FIG. 84 conceptually illustrates another software architecture for avideo conferencing and processing module of a dual camera mobile deviceof some embodiments.

FIG. 85 illustrates a user interface of some embodiments for amulti-participant video conference.

FIG. 86 illustrates another user interface of some embodiments for amulti-participant video conference.

FIG. 87 illustrates another user interface of some embodiments for amulti-participant video conference.

FIG. 88 conceptually illustrates an application programming interface(API) architecture of some embodiments.

FIG. 89 illustrates an architecture for a dual camera mobile computingdevice of some embodiments.

FIG. 90 conceptually illustrates a touch input/output (I/O) device ofsome embodiments.

FIG. 91 conceptually illustrates an example communication system of someembodiments.

FIG. 92 conceptually illustrates another example communication system ofsome embodiments.

DETAILED DESCRIPTION

In the following description, numerous details are set forth for purposeof explanation. However, one of ordinary skill in the art will realizethat the invention may be practiced without the use of these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order not to obscure the description of theinvention with unnecessary detail.

Some embodiments of the invention provide a mobile device with twocameras that can take pictures and videos. Examples of mobile devicesinclude mobile phones, smartphones, personal digital assistants (PDAs),laptops, tablet personal computers, or any other type of mobilecomputing device. As used in this document, pictures refer to stillpicture images that are taken by the camera one at a time in asingle-picture mode, or several at a time in a fast-action mode. Video,on the other hand, refers to a sequence of video images that arecaptured by a camera at a particular rate, which is often referred to asa frame rate. Typical frame rates for capturing video are 25 frames persecond (fps), 30 fps, and 60 fps. The cameras of the mobile device ofsome embodiments can capture video images (i.e., video frames) at theseand other frame rates.

The mobile device of some embodiments (1) can display the capturedpicture images and video images, (2) can store the captured images forlater transmission to another device, (3) can transmit the capturedimages to one or more devices during a real-time communication sessionbetween the users of the devices, and (4) can encode the captured imagesfor local storage or for transmission to another device.

One example of a real-time communication session that involves thetransmission of the captured video images is a video conference. In someembodiments, the mobile device can only transmit one camera's capturedvideo images at any given time during a video conference. In otherembodiments, however, the mobile device can transmit captured videoimages from both of its cameras simultaneously during a video conferenceor other real-time communication session.

The mobile devices of some embodiments generate composite displays thatinclude simultaneous display of multiple videos captured by multiplecameras of one or more devices. In some cases, the composite displaysplace the videos in adjacent display areas (e.g., in adjacent windows).FIG. 1 illustrates one such example of a composite display 100 thatincludes two adjacent display areas 105 and 110 that simultaneouslydisplay two videos captured by two cameras of one device or captured bytwo cameras of two different devices that are in a video conference.

In other cases, the composite display is a PIP display that includes atleast two display areas that show two different videos, where one of thedisplay areas is a background main display area and the other is aforeground inset display area that overlaps the background main displayarea. FIG. 2 illustrates one such example of a composite PIP display200. This composite PIP display 200 includes a background main displayarea 205 and a foreground inset display area 210 that overlaps thebackground main display area. The two display areas 205 and 210simultaneously display two videos captured by two cameras of one device,or captured by two cameras of two different devices that are in a videoconference. While the example composite PIP displays illustrated anddiscussed in this document are similar to the composite PIP display 200,which shows the entire foreground inset display area 210 within thebackground main display area 205, other composite PIP displays that havethe foreground inset display area 210 overlapping, but not entirelyinside, the background main display area 205 are possible.

In addition to transmitting video content during a video conference withanother device, the mobile device of some embodiments can transmit othertypes of content along with the conference's video content. One exampleof such other content includes low or high resolution picture imagesthat are captured by one of the device's cameras, while the device'sother camera is capturing a video that is used in the video conference.Other examples of such other content include (1) files and other contentstored on the device, (2) the screen display of the device (i.e., thecontent that is displayed on the device's screen), (3) content receivedfrom another device during a video conference or other real-timecommunication session, etc.

The mobile devices of some embodiments employ novel in-conferenceadjustment techniques for making adjustments during a video conference.For instance, while transmitting only one camera's captured video duringa video conference, the mobile device of some embodiments candynamically switch to transmitting the video captured by its othercamera. In such situations, the mobile device of some embodimentsnotifies any other device participating in the video conference of thisswitch so that this other device can provide a smooth transition on itsend between the videos captured by the two cameras.

In some embodiments, the request to switch cameras not only canoriginate on the “local” device that switches between its cameras duringthe video conference, but also can originate from the other “remote”device that is receiving the video captured by the local device.Moreover, allowing one device to direct another device to switch camerasis just one example of a remote control capability of the devices ofsome embodiments. Examples of other operations that can be directed to adevice remotely in some embodiments include exposure adjustmentoperations (e.g., auto-exposure), focus adjustment operations (e.g.,auto-focus), etc. Another example of a novel in-conference adjustmentthat can be specified locally or remotely is the identification of aregion of interest (ROI) in a captured video, and the use of this ROIidentification to modify the behavior of the capturing camera, to modifythe image processing operation of the device with the capturing camera,or to modify the encoding operation of the device with the capturingcamera.

Yet another example of a novel in-conference adjustment of someembodiments involves real-time modifications of composite video displaysthat are generated by the devices. Specifically, in some embodiments,the real-time modifications of the composite video displays involvemoving one or more of the display areas within a composite display inresponse to a user's selection and movement of the display areas. Someembodiments also rotate the composite display during a video conference,when the screen of the device that provides this composite displayrotates. Also, the mobile device of some embodiments allow the user ofthe device to flip the order of videos in a PIP display (i.e., to makethe video in the foreground inset display appear in the background maindisplay, while making the video in the background main display appear inthe foreground inset display).

Several more detailed embodiments are described below. Section Iprovides a description of the video processing architecture of someembodiments. Section II then describes the captured image processingunit of some embodiments. In some embodiments, this unit is thecomponent of the device that is responsible for processing raw imagescaptured by the cameras of the device.

Next, Section III describes the video conferencing architecture of someembodiments. This section also describes the video conference module ofsome embodiments, as well as several manners for setting up a singlecamera video conference. Section IV then describes in-conferenceadjustment and control operations of some embodiments. Section V thendescribes video conference features of embodiments that transmit anddisplay multiple videos from individual devices during a videoconference. Section VI next describes transmission of real-time videoalong with non real-time content during a video conference. Lastly,Section VII describes the hardware architecture of the dual cameradevice of some embodiments.

I. Video Capture and Processing

FIG. 3 conceptually illustrates a video processing and encoding module300 of a dual camera mobile device of some embodiments. In someembodiments, the module 300 processes images and encodes videos that arecaptured by the cameras of the dual camera mobile device. As shown inFIG. 3 , this module 300 includes a captured image processing unit(CIPU) driver 305, a media exchange module 310, an encoder driver 320,and a video processing module 325.

In some embodiments, the media exchange module 310 allows programs onthe device that are consumers and producers of media content to exchangemedia content and instructions regarding the processing of the mediacontent. In the video processing and encoding module 300, the mediaexchange module 310 of some embodiments routes instructions and mediacontent between the video processing module 325 and the CIPU driver 305,and between the video processing module 325 and the encoder driver 320.To facilitate the routing of such instructions and media content, themedia exchange module 310 of some embodiments provides a set ofapplication programming interfaces (APIs) for the consumers andproducers of media content to use. In some of such embodiments, themedia exchange module 310 is a set of one or more frameworks that ispart of an operating system running on the dual camera mobile device.One example of such a media exchange module 310 is the Core Mediaframework provided by Apple Inc.

The video processing module 325 performs image processing on the imagesand/or the videos captured by the cameras of the device. Examples ofsuch operations include exposure adjustment operations, focus adjustmentoperations, perspective correction, dynamic range adjustment, imageresizing, image compositing, etc. In some embodiments, some imageprocessing operations can also be performed by the media exchange module310. For instance, as shown in FIG. 3 , the media exchange module 310 ofsome embodiments performs a temporal noise reduction (TNR) operation(e.g., by TNR 315) that reduces noise in video images captured by thecameras of the device. Further examples of such image processingoperations of the video processing module 325 and the media exchangemodule 310 will be provided below.

Through the media exchange module 310, the video processing module 325interfaces with the CIPU driver 305 and the encoder driver 320, asmentioned above. The CIPU driver 305 serves as a communication interfacebetween a captured image processing unit (CIPU) 330 and the mediaexchange module 310. As further described below, the CIPU 330 is thecomponent of the dual camera device that is responsible for processingimages captured during image capture or video capture operations of thedevice's cameras. From the video processing module 325 through the mediaexchange module 310, the CIPU driver 305 receives requests for imagesand/or videos from one or both of the device's cameras. The CIPU driver305 relays such requests to the CIPU 330, and in response receives therequested images and/or videos from the CIPU 330, which the CIPU driver305 then sends to the video processing module 325 through the mediaexchange module 310. Through the CIPU driver 305 and the media exchangemodule 310, the video processing module 325 of some embodiments alsosends instructions to the CIPU 330 in order to modify some of itsoperations (e.g., to modify a camera's frame rate, exposure adjustmentoperation, focus adjustment operation, etc.).

The encoder driver 320 serves as a communication interface between themedia exchange module 310 and an encoder hardware 335 (e.g., an encoderchip, an encoding component on a system on chip, etc.). In someembodiments, the encoder driver 320 receives images and requests toencode the images from the video processing module 325 through the mediaexchange module 310. The encoder driver 320 sends the images to beencoded to the encoder 335, which then performs picture encoding orvideo encoding on the images. When the encoder driver 320 receivesencoded images from the encoder 335, the encoder driver 320 sends theencoded images back to the video processing module 325 through the mediaexchange module 310.

In some embodiments, the video processing module 325 can performdifferent operations on the encoded images that it receives from theencoder. Examples of such operations include storing the encoded imagesin a storage of the device, transmitting the encoded images in a videoconference through a network interface of the device, etc.

In some embodiments, some or all of the modules of the video processingand encoding module 300 are implemented as part of an operating system.For example, some embodiments implement all four components 305, 310,320, and 325 of this module 300 as part of the operating system of thedevice. Other embodiments implement the media exchange module 310, theCIPU driver 305, and the encoder driver 320 as part of the operatingsystem of the device, while having the video processing module 325 as anapplication that runs on the operating system. Still, otherimplementations of the module 300 are possible.

The operation of the video processing and encoding module 300 during avideo capture session will now be described. To start a video capturesession, the video processing module 325 initializes several componentsthat are needed for the video capture session. In some embodiments,these components include (1) the CIPU 330, (2) a scaling and compositingmodule (not shown) of the video processing module 325, (3) an imageprocessing module (not shown) of the video processing module 325, and(4) the encoder 335. Also, the video processing module 325 of someembodiments initializes a network manager (not shown) when it isparticipating in a video conference.

Through the media exchange module 310 and the CIPU driver 305, the videoprocessing module sends its initialization request to the CIPU 330, inorder to have one or both of the cameras of the device start videocapturing. In some embodiments, this request specifies a particularframe rate, exposure level, and scaling size for each camera that needsto capture a video. In response to this request, the CIPU 330 starts toreturn video images from the requested cameras at the specified rate(s),exposure level(s), and scaling size(s). These video images are returnedto the video processing module 325 through the CIPU driver 305 and themedia exchange module 310, which, as mentioned above, performs TNRoperations on the video images before supplying them to the videoprocessing module 325. At the video processing module 325, the videoimages are stored in a buffer (not shown) for additional imageprocessing.

The image processing module of the video processing module 325 retrievesthe video images stored in the buffer for additional video processing.The scaling and compositing module then retrieves the processed videoimages in order to scale them if necessary for real time display on thedisplay screen of the device. In some embodiments, this module createscomposite images from the images captured by two cameras of the deviceor from images captured by the camera(s) of the device along with thecamera(s) of another device during a video conference in order toprovide a real-time display of the captured video images on the deviceor to create a composite video image for encoding.

The processed and/or composited video images are supplied to the encoder335 through the encoder driver 320 and the media exchange module 310.The encoder 335 then encodes the video images. The encoded images arethen returned to the video processing module 325 (again through theencoder driver 320 and the media exchange module 310) for storage on thedevice or for transmission during a video conference. When the device isparticipating in a video conference, the network manager (that wasinitialized by the video processing module 325) then retrieves theseencoded images, packetizes them and transmits them to one or more otherdevices through a network interface (not shown) of the device.

II. Captured Image Processing

A. Single Pipeline

The images captured by cameras of the dual camera mobile device of someembodiments are raw, unprocessed images. These images require conversionto a particular color space before the images can be used for otheroperations such as transmitting the images to another device (e.g.,during a video conference), storing the images, or displaying theimages. In addition, the images captured by the cameras may need to beprocessed to correct errors and/or distortions and to adjust the images'color, size, etc. Accordingly, some embodiments perform severalprocessing operations on the images before storing, transmitting, anddisplaying such images. Part of the processing of such images isperformed by the CIPU 330.

One example of such a CIPU is illustrated in FIG. 4 . Specifically, thisfigure conceptually illustrates a captured image processing unit (CIPU)400 of some embodiments. This CIPU 400 includes a single processingpipeline 485 that either processes images from only one of the device'scameras at a time, or processes images from both of the device's camerassimultaneously in a time-division multiplex fashion (i.e., in a timeinterleaved manner). The CIPU 400's processing pipeline 485 can beconfigured differently to address differing characteristics and/oroperational settings of the different cameras. Examples of differentcamera characteristics in some embodiments include differentresolutions, noise sensors, lens types (fixed or zoom lens), etc. Also,examples of different operational settings under which the device canoperate the cameras in some embodiments include image resolution size,frame rate, zoom level, exposure level, etc.

As shown in FIG. 4 , the CIPU 400 includes a sensor module 415, aline/frame buffer 417, a bad pixel correction (BPC) module 420, a lensshading (LS) module 425, a demosaicing module 430, a white balance (WB)module 435, a gamma module 440, a color space conversion (CSC) module445, a hue, saturation, and contrast (HSC) module 450, a scaler module455, a filter module 460, a statistics engine 465, two sets of registers470, and a controller module 475. In some embodiments, all of themodules of the CIPU 400 are implemented in hardware (e.g., an ASIC,FPGA, a SOC with a microcontroller, etc.), while in other embodiments,some or all of the modules of the CIPU 400 are implemented in software.

As shown in FIG. 4 , the sensor module 415 communicatively couples totwo pixel arrays 410 a and 410 b and two sets of sensors 405 a and 405 bof two cameras of the device. In some embodiments, this communicativecoupling is facilitated through each camera sensor's mobile industryprocessor interface (MIPI).

Through this communicative coupling, the sensor module 415 can forwardinstructions to the cameras to control various aspects of each camera'soperations such as its power level, zoom level, focus, exposure level,etc. In some embodiments, each camera has four operational power modes.In the first operational power mode, the camera is powered off. For thesecond operational power mode, the camera is powered on, but it is notyet configured. In the third operational power mode, the camera ispowered on, the camera's sensor is configured, and the camera sensor'spixels are collecting photons and converting the collected photons todigital values. However, the camera sensor is not yet sending images tothe sensor module 415. Finally, in the fourth operational power mode,the camera is in the same operational power mode as the third power modeexcept the camera is now sending images to the sensor module 415.

During the operation of the device, the cameras may switch from oneoperational power mode to another any number of times. When switchingoperational power modes, some embodiments require the cameras to switchoperational power modes in the order described above. Therefore, inthose embodiments, a camera in the first operational power mode can onlyswitch to the second operational power mode. When the camera is in thesecond operational power mode, it can switch to the first operationalpower mode or to the third operational power mode. Similarly, the cameracan switch from the third operational power mode to the secondoperational power mode or the fourth operation power mode. When thecamera is in the fourth operational power mode, it can only switch backto the third operational power mode.

Moreover, switching from one operational power mode to the next or theprevious operational power mode takes a particular amount of time. Thus,switching between two or three operational power modes is slower thanswitching between one operational power mode. The different operationalpower modes also consume different amounts of power. For instance, thefourth operational power mode consumes the most amount of power, thethird operational power mode consumes more power than the first andsecond, and the second operational power mode consumes more than thefirst. In some embodiments, the first operational power mode does notconsume any power.

When a camera is not in the fourth operational power mode capturingimages, the camera may be left in one of the other operational powermodes. Determining the operational mode in which to leave the unusedcamera depends on how much power the camera is allowed to consume andhow fast the camera may need to respond to a request to start capturingimages. For example, a camera configured to operate in the thirdoperational power mode (e.g., standby mode) consumes more power than acamera configured to be in the first operational power mode (i.e.,powered off). However, when the camera is instructed to capture images,the camera operating in the third operational power mode can switch tothe fourth operational power mode faster than the camera operating inthe first operational power mode. As such, the cameras can be configuredto operate in the different operational power modes when not capturingimages based on different requirements (e.g., response time to a requestto capture images, power consumption).

Through its communicative coupling with each camera, the sensor module415 can direct one or both sets of camera sensors to start capturingimages when the video processing module 325 requests one or both camerasto start capturing images and the sensor module 415 receives thisrequest through the controller module 475, as further described below.Bayer filters are superimposed over each of the camera sensors and thuseach camera sensor outputs Bayer pattern images, which are stored in thepixel array associated with each camera sensor. A Bayer pattern image isan image where each pixel only stores one color value: red, blue, orgreen.

Through its coupling with the pixel arrays 410 a and 410 b, the sensormodule 415 retrieves raw Bayer pattern images stored in the camera pixelarrays 410 a and 410 b. By controlling the rate at which the sensormodule 415 retrieves images from a camera's pixel array, the sensormodule 415 can control the frame rate of the video images that are beingcaptured by a particular camera. By controlling the rate of its imageretrieval, the sensor module 415 can also interleave the fetching ofimages captured by the different cameras in order to interleave the CIPUprocessing pipeline 485's image processing of the captured images fromthe different cameras. The sensor module 415's control of its imageretrieval is further described below in sub-sections II.A.1 and II.A.2.

The sensor module 415 stores image lines (i.e., rows of pixels of animage) in the line/frame buffer 417, which the sensor module 415retrieves from the pixel arrays 410 a and 410 b. Each image line in theline/frame buffer 417 is processed through the CIPU processing pipeline485. As shown in FIG. 4 , the CIPU processing pipeline 485 is formed bythe BPC module 420, the LS module 425, the demosaicing module 430, theWB module 435, the gamma module 440, the CSC module 445, the HSC module450, the scaler module 455, and the filter module 460. In someembodiments, the CIPU processing pipeline 485 processes images from theline/frame buffer 417 on a line-by-line (i.e., row-by-row) basis whilein other embodiments the CIPU processing pipeline 485 processes entireimages from the line/frame buffer 417 on a frame-by-frame basis.

In the exemplary pipeline illustrated in FIG. 4 , the BPC module 420 isthe module that retrieves the images from the line/frame buffer 417.This module performs a bad-pixel removal operation that attempts tocorrect bad pixels in the retrieved images that might have resulted fromone or more of the camera sensors being defective (e.g., the defectivephoto sensors do not sense light at all, sense light incorrectly, etc.).In some embodiments, the BPC module 420 detects bad pixels by comparinga particular pixel in an image with one or more neighboring pixels inthe image. If the difference between the value of the particular pixeland the values of the neighboring pixels is greater than a thresholdamount, the particular pixel's value is replaced by the average ofseveral neighboring pixels' values that are of the same color (i.e.,red, green, and blue) as the particular pixel.

The operation of the BPC module 420 is in part controlled by the valuesstored for this module in the two sets of registers 470 of the CIPU 400.Specifically, to process the images captured by the two differentcameras of the device, some embodiments configure the CIPU processingpipeline 485 differently for each camera, as mentioned above. The CIPUprocessing pipeline 485 is configured for the two different cameras bystoring two different sets of values in the two different sets ofregisters 470 a (Ra) and 470 b (Rb) of the CIPU 400. Each set ofregisters 470 includes one register (Ra or Rb) for each of the modules420-460 within the CIPU processing pipeline 485. Each register in eachregister set stores a set of values that defines one processing pipelinemodule's operation. Accordingly, as shown in FIG. 4 , the register set470 a is for indicating the mode of operation of each processingpipeline module for one camera (camera A) of the dual camera mobiledevice, while the register set 470 b is for indicating the mode ofoperation of each module for the other camera (camera B) of the dualcamera mobile device.

One example of configuring the CIPU processing pipeline 485 differentlyfor each camera is to configure the modules of the CIPU processingpipeline 485 to process different sized images. For instance, if thecamera sensor 405 a is 640×480 pixels and the camera sensor 405 b is2048×1536 pixels, the set of registers 470 a is configured to storevalues that instruct the modules of the CIPU processing pipeline 485 toprocess 640×480 pixel images and the set of registers 470 b isconfigured to store values that instruct the modules of the CIPUprocessing pipeline 485 to process 2048×1536 pixel images.

In some embodiments, different processing pipeline configurations (i.e.,register values) are stored in different profile settings. In some ofsuch embodiments, a user of the mobile device is allowed to select oneof the profile settings (e.g., through a user interface displayed on themobile device) to set the operation of a camera(s). For example, theuser may select a profile setting for configuring a camera to capturehigh resolution video, a profile setting for configuring the same camerato capture low resolution video, or a profile setting for configuringboth cameras to capture high resolution still images. Differentconfigurations are possible, which can be stored in many differentprofile settings. In other of such embodiments, instead of allowing theuser to select a profile setting, a profile setting is automaticallyselected based on which application or activity the user selects. Forinstance, if the user selects a video conferencing application, aprofile that configures both cameras to capture video is automaticallyselected, if the user selects a photo application, a profile thatconfigures one of the cameras to capture still images is automaticallyselected, etc.

After the BPC module 420, the LS module 425 receives thebad-pixel-corrected images. The LS module 425 performs a lens shadingcorrection operation to correct for image defects that are caused bycamera lenses that produce light falloff effects (i.e., light is reducedtowards the edges of the camera sensor). Such effects cause images to beunevenly illuminated (e.g., darker at corners and/or edges). To correctthese image defects, the LS module 425 of some embodiments estimates amathematical model of a lens' illumination fall-off. The estimated modelis then used to compensate the lens fall-off of the image to evenlyilluminate unevenly illuminated portions of the image. For example, if acorner of the image is half the brightness of the center of the image,the LS module 425 of some embodiments multiplies the corner pixels valueby two in order to produce an even image.

The demosaicing module 430 performs a demosaicing operation to generatefull color images from images of sampled colors. As noted above, thecamera sensors output Bayer pattern images, which are incomplete becauseeach pixel of a Bayer pattern image stores only one color value. Thedemosaicing module 430 reconstructs a red, green, blue (RGB) image froma Bayer pattern image by interpolating the color values for each set ofcolors in the Bayer pattern image.

The WB module 435 performs a white balance operation on the RGB imagesreceived from the demosaicing module 430 so that the colors of thecontent of the images are similar to the colors of such contentperceived by the human eye in real life. The WB module 435 adjusts thewhite balance by adjusting colors of the images to render neutral colors(e.g., gray, white, etc.) correctly. For example, an image of a piece ofwhite paper under an incandescent light may appear yellow whereas thehuman eye perceives the piece of paper as white. To account for thedifference between the color of the images that the sensor captures andwhat the human eye perceives, the WB module 435 adjusts the color valuesof the image so that the captured image properly reflects the colorsperceived by the human eye.

The statistics engine 465 collects image data at various stages of theCIPU processing pipeline 485. For example, FIG. 4 shows that thestatistics engine 465 collects image data after the LS module 425, thedemosaicing module 430, and the WB module 435. Different embodimentscollect data from any number of different stages of the CIPU processingpipeline 485. The statistics engine 465 processes the collected data,and, based on the processed data, adjusts the operations of the camerasensors 405 a and 405 b through the controller module 475 and the sensormodule 415. Examples of such operations include exposure and focus.Although FIG. 4 shows the statistics engine 465 controlling the camerasensors 405 a and 405 b through the controller module 475, otherembodiments of the statistics engine 465 control the camera sensorsthrough just the sensor module 415.

The processed data can also be used to adjust the operations of variousmodules of the CIPU 400. For instance, the statistics engine 465 of someembodiments adjusts the operations of the WB module 435 based on datacollected after the WB module 435. In some of such embodiments, thestatistics engine 465 provides an automatic white balance (AWB) functionby using the processed data to adjust the white balancing operation ofthe WB module 435. Other embodiments can use processed data collectedfrom any number of stages of the CIPU processing pipeline 485 to adjustthe operations of any number of modules within the CIPU processingpipeline 485. Further, the statistics engine 465 can also receiveinstructions from the controller module 475 to adjust the operations ofone or more modules of the CIPU processing pipeline 485.

After receiving the images from the WB module 435, the gamma module 440performs a gamma correction operation on the image to code and decodeluminance or tristimulus values of the camera system. The gamma module440 of some embodiments corrects gamma by converting a 10-12 bit linearsignal into an 8 bit non-linear encoding in order to correct the gammaof the image. Some embodiments correct gamma by using a lookup table.

The CSC module 445 converts the image received from the gamma module 440from one color space to another color space. Specifically, the CSCmodule 445 converts the image from an RGB color space to a luminance andchrominance (YUV) color space. However, other embodiments of the CSCmodule 445 can convert images from and to any number of color spaces.

The HSC module 450 may adjust the hue, saturation, contrast, or anycombination thereof of the images received from the CSC module 445. TheHSC module 450 may adjust these properties to reduce the noise orenhance the images, for example. For instance, the saturation of imagescaptured by a low-noise camera sensor can be increased to make theimages appear more vivid. In contrast, the saturation of images capturedby a high-noise camera sensor can be decreased to reduce the color noiseof such images.

After the HSC module 450, the scaler module 455 may resize images toadjust the pixel resolution of the image or to adjust the data size ofthe image. The scaler module 455 may also reduce the size of the imagein order to fit a smaller display, for example. The scaler module 455can scale the image a number of different ways. For example, the scalermodule 455 can scale images up (i.e., enlarge) and down (i.e., shrink).The scaler module 455 can also scale images proportionally or scaleimages anamorphically.

The filter module 460 applies one or more filter operations to imagesreceived from the scaler module 455 to change one or more attributes ofsome or all pixels of an image. Examples of filters include a low-passfilter, a high-pass filter, a band-pass filter, a bilateral filter, aGaussian filter, among other examples. As such, the filter module 460can apply any number of different filters to the images.

The controller module 475 of some embodiments is a microcontroller thatcontrols the operation of the CIPU 400. In some embodiments, thecontroller module 475 controls (1) the operation of the camera sensors(e.g., exposure level) through the sensor module 415, (2) the operationof the CIPU processing pipeline 485, (3) the timing of the CIPUprocessing pipeline 485 (e.g., when to switch camera sensors, when toswitch registers, etc.), and (4) a flash/strobe (not shown), which ispart of the dual camera mobile device of some embodiments.

Some embodiments of the controller module 475 process instructionsreceived from the statistics engine 465 and the CIPU driver 480. In someembodiments, the instructions received from the CIPU driver 480 areinstructions from the dual camera mobile device (i.e., received from thelocal device) while in other embodiments the instructions received fromthe CIPU driver 480 are instructions from another device (e.g., remotecontrol during a video conference). Based on the processed instructions,the controller module 475 can adjust the operation of the CIPU 400 byprogramming the values of the registers 470. Moreover, the controllermodule 475 can dynamically reprogram the values of the registers 470during the operation of the CIPU 400.

As shown in FIG. 4 , the CIPU 400 includes a number of modules in theCIPU processing pipeline 485. However, one of ordinary skill willrealize that the CIPU 400 can be implemented with just a few of theillustrated modules or with additional and different modules. Inaddition, the processing performed by the different modules can beapplied to images in sequences different from the sequence illustratedin FIG. 4 .

An example operation of the CIPU 400 will now be described by referenceto FIG. 4 . For purposes of explanation, the set of registers Ra is usedfor processing images captured by camera sensor 405 a of the dual cameramobile device and the set of registers Rb is used for processing imagescaptured by camera sensor 405 b of the dual camera mobile device. Thecontroller module 475 receives instructions from the CIPU driver 480 toproduce images captured by one of the cameras of the dual camera mobiledevice.

The controller module 475 then initializes various modules of the CIPUprocessing pipeline 485 to process images captured by one of the camerasof the dual camera mobile device. In some embodiments, this includes thecontroller module 475 checking that the correct set of registers of theregisters 470 are used. For example, if the CIPU driver 480 instructsthe controller module 475 to produce images captured by the camerasensor 405 a, the controller module 475 checks that the set of registersRa is the set of registers from which the modules of the CIPU 400 read.If not, the controller module 475 switches between the sets of registersso that the set of registers Ra is the set that is read by the modulesof the CIPU 400.

For each module in the CIPU processing pipeline 485, the mode ofoperation is indicated by the values stored in the set of registers Ra.As previously mentioned, the values in the set of registers 470 can bedynamically reprogrammed during the operation of the CIPU 400. Thus, theprocessing of one image can differ from the processing of the nextimage. While the discussion of this example operation of the CIPU 400describes each module in the CIPU 400 reading values stored in registersto indicate the mode of operation of the modules, in somesoftware-implemented embodiments, parameters are instead passed to thevarious modules of the CIPU 400.

In some embodiments, the controller module 475 initializes the sensormodule 415 by instructing the sensor module 415 to delay a particularamount of time after retrieving an image from the pixel array 410 a. Inother words, the controller module 475 instructs the sensor module 415to retrieve the images from the pixel array 410 a at a particular rate.

Next, the controller module 475 instructs the camera sensor 405 athrough the sensor module 415 to capture images. In some embodiments,the controller module 475 also provides exposure and other cameraoperation parameters to the camera sensor 405 a. In other embodiments,the camera sensor 405 a uses default values for the camera sensoroperation parameters. Based on the parameters, the camera sensor 405 acaptures a raw image, which is stored in the pixel array 410 a. Thesensor module 415 retrieves the raw image from the pixel array 410 a andsends the image to the line/frame buffer 417 for storage before the CIPUprocessing pipeline 485 processing the image.

Under certain circumstances, images may be dropped by the line/framebuffer 417. When the camera sensors 405 a and/or 405 b are capturingimages at a high rate, the sensor module 415 may receive and storeimages in the line/frame buffer 417 faster than the BPC module 420 canretrieve the images from the line/frame buffer 417 (e.g., capturing highframe-rate video), and the line/frame buffer 417 will become full. Whenthis happens, the line/frame buffer 417 of some embodiments drops images(i.e., frames) based on a first in, first out basis. That is, when theline/frame buffer 417 drops an image, the line/frame buffer 417 dropsthe image that was received before all the other images in theline/frame buffer 417.

The processing of the image by the CIPU processing pipeline 485 startsby the BPC module 420 retrieving the image from the line/frame buffer417 to correct any bad pixels in the image. The BPC module 420 thensends the image to the LS module 425 to correct for any unevenillumination in the image. After the illumination of the image iscorrected, the LS module 425 sends the image to the demosaicing module430 where it processes the raw image to generate an RGB image from theraw image. Next, the WB module 435 receives the RGB image from thedemosaicing module 430 and adjusts the white balance of the RGB image.

As noted above, the statistics engine 465 may have collected some dataat various points of the CIPU processing pipeline 485. For example, thestatistics engine 465 collects data after the LS module 425, thedemosaicing module 430, and the WB module 435 as illustrated in FIG. 4 .Based on the collected data, the statistics engine 465 may adjust theoperation of the camera sensor 405 a, the operation of one or moremodules in the CIPU processing pipeline 485, or both, in order to adjustthe capturing of subsequent images from the camera sensor 405 a. Forinstance, based on the collected data, the statistics engine 465 maydetermine that the exposure level of the current image is too low andthus instruct the camera sensor 405 a through the sensor module 415 toincrease the exposure level for subsequently captured images. Thus, thestatistics engine 465 of some embodiments operates as a feedback loopfor some processing operations.

After the WB module 435 adjusts the white balance of the image, it sendsthe image to the gamma module 440 for gamma correction (e.g., adjustingthe gamma curve of the image). The CSC module 445 receives thegamma-corrected image from the gamma module 440 and performs color spaceconversion. In this example, the CSC module 445 converts the RGB imageto a YUV image. In other words, the CSC module 445 converts an imagethat is represented in an RGB color space to an image that isrepresented in a YUV color space. The HSC module 450 receives the YUVimage from the CSC module 445 and adjusts the hue, saturation, andcontrast attributes of various pixels in the image. After the HSC module450, the scaler module 455 resizes the image (e.g., enlarging orshrinking the image). The filter module 460 applies one or more filterson the image after receiving the image from the scaler module 455.Finally, the filter module 460 sends the processed image to the CIPUdriver 480.

In this example of the operation of the CIPU 400 described above, eachmodule in the CIPU processing pipeline 485 processed the image in somemanner. However, other images processed by the CIPU 400 may not requireprocessing by all the modules of the CIPU processing pipeline 485. Forexample, an image may not require white balance adjustment, gammacorrection, scaling, or filtering. As such, the CIPU 400 can processimages any number of ways based on a variety of received input such asinstructions from the CIPU driver 480 or data collected by the statisticengine 465, for example.

Different embodiments control the rate at which images are processed(i.e., frame rate) differently. One manner of controlling the frame rateis through manipulation of vertical blanking intervals (VBI). For someembodiments that retrieve image lines for processing images on aline-by-line basis, a VBI is the time difference between retrieving thelast line of an image of a video captured by a camera of the dual cameramobile device from a pixel array and retrieving the first line of thenext image of the video from the pixel array. In other embodiments, aVBI is the time difference between retrieving one image of a videocaptured by a camera of the dual camera mobile device from a pixel arrayand retrieving the next image of the video the pixel array.

One example where VBI can be used is between the sensor module 415 andthe pixel arrays 410 a and 410 b. For example, some embodiments of thesensor module 415 retrieve images from the pixel arrays 410 a and 410 bon a line-by-line basis and other embodiments of the sensor module 415retrieve images from the pixel arrays 410 a and 410 b on animage-by-image basis. Thus, the frame rate can be controlled byadjusting the VBI of the sensor module 415: increasing the VBI reducesthe frame rate and decreasing the VBI increases the frame rate.

1. Use of VBI for Single Camera: Frame Rate Control

FIG. 5 conceptually illustrates examples of different frame rates 505,510, and 515 based on different VBIs. Each sequence shows an image,which is captured by one of the cameras of the dual camera mobiledevice, of a person holding a guitar at various time instances 525-555along timeline 520. In addition, the time between each time instance525-555 is the same and will be referred to as one time unit. Forpurposes of explanation, FIG. 5 will now be described by reference tothe sensor module 415 and the pixel array 410 a of FIG. 4 . As such,each image represents a time instance along the timeline 520 at whichthe sensor module 415 retrieves an image from the pixel array 410 a.

In the example frame rate 505, the VBI of the sensor module 415 for thepixel array 410 a is set to three time units (e.g., by the controllermodule 475). That is, the sensor module 415 retrieves an image from thepixel array 410 a every third time instance along the timeline 520. Asshown in the example frame rate 505, the sensor module 415 retrieves animage at the time instances 525, 540, and 555. Thus, the example framerate 505 has a frame rate of one image per three time units.

The example frame rate 510 is similar to the example frame rate 505except the VBI is set to two time units. Thus, the sensor module 415retrieves an image from the pixel array 410 a every second time instancealong the timeline 520. The example frame rate 510 shows the sensormodule 415 retrieving an image from the pixel array 410 a at the timeinstances 525, 535, 545, and 555. Since the VBI of the example framerate 510 is less than the VBI of the example frame rate 505, the framerate of the example frame rate 510 is higher than the frame rate of theexample frame rate 505.

The example frame rate 515 is also similar to the example frame rate 505except the VBI of the sensor module 415 for the pixel array 410 a is setto one time unit. Therefore, the sensor module 415 is instructed toretrieve an image from the pixel array 410 a every time instance alongthe timeline 520. As illustrated, the sensor module 415 retrieves animage from the pixel array 410 a at the time instances 525-555. The VBIof the example frame rate 515 is less than the VBIs of the example framerates 505 and 510. Therefore, the frame rate of the example frame rate515 is higher than the example frame rates 505 and 510.

2. Use of VBI for Two Cameras

Some embodiments may wish to operate both cameras of the dual cameramobile device at the same time (e.g., transmit videos from both camerasduring a video conference). Different embodiments of the dual cameramobile device that include a single processing pipeline providedifferent mechanisms for simultaneously operating both cameras of thedual camera mobile device.

One such mechanism is interleaving the processing of images captured byboth cameras by controlling each camera's VBI. That is, one or moreimages captured by one camera are captured and processed during the VBIof the other camera and vice versa. Since the CIPU 400 described abovehas a single processing pipeline 485, this mechanism can be implementedin the CIPU 400 of some embodiments. In such embodiments, the sensormodule 415 retrieves an image from one of the pixel arrays 410 a and 410b and the retrieved image is processed by the CIPU 400 during the sensormodule 415's VBI for the other pixel array.

The sensor module 415's VBI for each pixel array can be set to aparticular value. However, in some embodiments, the VBI is not set to avalue that is less than the time it takes for the CIPU 400 to retrieveand process one image. Some embodiments set the sensor module 415's VBIfor each pixel array to the same value. For example, when the sensormodule 415's VBI for each pixel array is set to the same value, thesensor module 415 alternately retrieves images from the pixel arrays 410a and 410 b. Other embodiments set the sensor module 415's VBI for eachpixel array to different values. In some of such embodiments, the sensormodule 415's VBI for one pixel array is set to a multiple of the sensormodule 415's VBI for the other pixel array. For instance, the sensormodule 415's VBI for one pixel array is set to 2 units of time, and thesensor module 415's VBI for the other pixel array is set to 4 units oftime. In this example, the sensor module 415 retrieves two images fromthe one pixel array for every one image the sensor module 415 retrievesfrom the other pixel array.

FIG. 6 conceptually illustrates examples of different interleaving framerates 605, 610, and 615 based on different VBIs. FIG. 6 is similar toFIG. 5 except FIG. 6 includes thirteen time instances 625-685 alongtimeline 620. In addition, the image of a person holding the guitarrepresents a time instance along the timeline 620 at which the image isretrieved from one pixel array while the image of the person wearing anacademic cap (i.e., a mortarboard) represents a time instance along thetimeline 620 at which the image is retrieved from the other pixel array.

For purposes of explanation, the image of the person holding the guitaris assumed to have been captured by the camera sensor 405 a of the dualcamera mobile device, and the image of the person wearing the academiccap is assumed to have been captured by the camera sensor 405 b of thedual camera mobile device. Moreover, FIG. 6 will now be described byreference to the sensor module 415 and the pixel arrays 410 a and 410 bof FIG. 4 .

In the example interleaving frame rate 605, the sensor module 415's VBIfor both the pixel array 410 a and the pixel array 410 b is set to twotime units. As illustrated in the example interleaving frame rate 605,the sensor module 415 retrieves an image from the pixel array 410 a atthe time instances 625, 635, 645, 655, 665, 675, and 685 along thetimeline 620, and the sensor module 415 retrieves an image from thepixel array 410 b at the time instances 630, 640, 650, 660, 670, and 680along the timeline 620. That is, the sensor module 415 alternatelyretrieves an image from a pixel array every time unit.

The example interleaving frame rate 610 is similar to the exampleinterleaving frame rate 605 except the sensor module 415's VBI for boththe pixel array 410 a and the pixel array 410 b is set to four timeunits. The example interleaving frame rate 610 shows the sensor module415 retrieving an image from the pixel array 410 a at the time instances625, 645, 665, and 685 along the timeline 620, and the sensor module 415retrieves an image from the pixel array 410 b at the time instances 635,655, and 675 along the timeline 620. Because the VBI of the exampleinterleaving frame rate 610 is greater than the VBI of the exampleinterleaving frame rate 605, the frame rate of the example interleavingframe rate 610 is lower than the frame rate of the example interleavingframe rate 605.

The example interleaving frame rate 615 is also similar to the exampleinterleaving frame rate 605 except the sensor module 415's VBI for boththe pixel array 410 a and the pixel array 410 b is set to six timeunits. As shown in FIG. 6 , the sensor module 415 retrieves an imagefrom the pixel array 410 a at the time instances 625, 655, and 685 alongthe timeline 620, and the sensor module 415 retrieves an image from thepixel array 410 b at the time instances 640 and 670 along the timeline620. The VBI of the example interleaving frame rate 615 is greater thanthe VBIs of the example interleaving frame rates 605 and 610. Thus, theframe rate of the example interleaving frame rate 615 is lower than theexample interleaving frame rates 605 and 610.

B. Multiple Pipelines

FIG. 7 conceptually illustrates another captured image processing unit(CIPU) 700 of some embodiments. The CIPU 700 performs the same functionsas the CIPU 400 described above except the CIPU 700 is implemented bytwo frontend processing pipelines, a storage, and a backend processingpipeline instead of a single processing pipeline. As such, thedescription of the functions of the CIPU 700 will be described byreference to the modules of the CIPU 400.

As shown, the CIPU 700 includes a frontend processing pipeline 715 forthe camera sensor 405 a and the pixel array 410 a, a frontend processingpipeline 720 for the camera sensor 405 b and the pixel array 410 b, astorage 725, a controller module 730, and a backend processing pipeline735. The camera sensors 405 a and 405 b of some embodiments are sensorsof the cameras of the dual camera mobile device.

The frontend processing pipelines 715 and 720 of some embodimentsperform a portion of the CIPU 400's image processing. As such, differentembodiments can include a different number of modules of the CIPU 400.For example, each of the frontend processing pipelines 715 and 720 ofsome embodiments includes the sensor module 415, the BPC module 420, theLS module 425, the demosaicing module 430, the WB module 435, and thestatistics engine 465 of the CIPU 400.

Although the frontend processing pipelines 715 and 720 perform the sametype of image processing by virtue of having the same modules, eachmodule of each of the frontend processing pipelines 715 and 720 can beconfigured differently through different register values as describedabove for the CIPU 400. Moreover, since each of the camera sensors 405 aand 405 b has its own frontend processing pipeline, the frontendprocessing pipelines 715 and 720 can process images independently ofeach other. For instance, the frontend processing pipelines 715 and 720can process images in parallel (i.e., at the same time), at differenttimes, and at different rates.

In some embodiments, each of the front end processing pipelines 715 and720 can retrieve images from its corresponding camera sensor and pixelarray. For instance, the frontend processing pipeline 715 retrievesimages captured by the camera sensor 405 a from the pixel array 410 aand the frontend processing pipeline 720 receives images captured by thecamera sensor 405 b from the pixel array 410 b. When one of the frontendprocessing pipelines 715 and 720 retrieves an image from itscorresponding camera sensor and pixel array, that frontend processingpipeline processes the image and sends the processed image to thestorage 725. Also, each of the frontend processing pipelines 715 and 720communicates with the controller module 730 as described above (e.g.,through the statistics engine of each frontend processing pipeline).

The storage 725 of some embodiments stores partially processed imagesfor the backend processing pipeline 735 to finish processing. In theseembodiments, the storage 725 receives partially processed images fromthe frontend processing pipelines 715 and 720 and sends the partiallyprocessed images to the backend processing pipeline 735. Someembodiments implement the storage 725 as volatile storage (e.g., randomaccess memory (RAM)) while other embodiments implement the storage 725as non-volatile storage (e.g. flash memory, hard disk, optical disk,etc.). Furthermore, the storage 725 of some embodiments is internalstorage (e.g., RAM) while the storage 725 of other embodiments isexternal storage (e.g., a compact flash (CF) card, a secure digital (SD)card, etc.).

Some embodiments of the backend processing pipeline 735 perform aportion of the CIPU 700's image processing. In some embodiments, thebackend processing pipeline 735 includes the modules of the CIPU 400that the frontend processing pipelines 715 and 720 do not include. Forinstance, referring to the above example, the backend processingpipeline 735 would include the CSC module 445, the gamma module 440, theHSC module 450, the scaler module 455, and the filter module 460 of theCIPU 400. As such, the backend processing pipeline 735 of suchembodiments performs the remaining image processing of the CIPU 400 thatthe frontend processing pipelines 715 and 720 do not perform.Accordingly, the backend processing pipeline 735 retrieves partiallyprocessed images from the storage 725 and performs the remaining imageprocessing on the partially processed images. After processing theimages, the backend processing pipeline 735 sends the processed imagesto the CIPU driver 480.

The controller module 730 performs the same functions described above byreference to FIG. 4 . As shown in FIG. 7 , the controller module 730interacts with the frontend processing pipelines 715 and 720 and thebackend processing pipeline 735. In some embodiments, the controllermodule 730 is included in the backend processing pipeline 735 while inother embodiments the controller module 730 is included in one of thefrontend processing pipelines 715 and 720.

The operation of the CIPU 700 will now be described by reference to thecamera sensors 405 a and 405 b, the pixel arrays 410 a and 410 b, thefrontend processing pipelines 715 and 720, the storage 725, and thebackend processing pipeline 735 that are illustrated in FIG. 7 . Whenone of the frontend processing pipelines 715 and 720 retrieves an imagefrom its corresponding camera sensor and pixel array, the frontendprocessing pipeline processes the image and sends the partiallyprocessed image to the storage 725. For instance, the frontendprocessing pipeline 715 may retrieve an image captured by the camerasensor 405 a from the pixel array 410 a or the frontend processingpipeline 720 may retrieve an image captured by the camera sensor 405 bfrom the pixel array 410 b. As noted above, each frontend processingpipeline 715 and 720 can process images in parallel.

The backend processing pipeline 735 retrieves the partially processedimage from the storage 725 and processes the partially processed imageto complete the image processing of the image. In some embodiments, thebackend processing pipeline 735 retrieves and processes images stored inthe storage 725 on a first in, first out basis. In other words, aparticular image in the storage 725 is processed after all images thatwere received and stored in the storage 725 before the particular image,but the particular image is processed before images that were receivedand stored in the storage 725 after the particular image. After thebackend processing pipeline 735 processes the image, it sends theprocessed image to the CIPU driver 480.

FIG. 8 conceptually illustrates another captured image processing unit(CIPU) 800 of some embodiments. The CIPU 800 performs the same functionsas the CIPU 400 described above except the CIPU 800 is implemented bytwo separate processing pipelines with each camera sensor having its ownseparate processing pipeline. As such, the description of the functionsof the CIPU 800 will be described by reference to the modules of theCIPU 400.

As shown, the CIPU 800 includes a processing pipeline 815 for the camerasensor 405 a and the pixel array 410 a and a processing pipeline 820 forthe camera sensor 405 b and the pixel array 410 b. Each of theprocessing pipelines 815 and 820 of some embodiments includes all themodules included in the CIPU 400. Therefore, the operation of each ofthe processing pipelines 815 and 820 of these embodiments is the same asthe operation of the CIPU 400.

Since each of the camera sensors 405 a and 405 b has its own processingpipeline, the processing pipelines 815 and 820 can process imagesindependently of each other. For example, the processing pipelines 815and 820 can process images in parallel (i.e., at the same time), atdifferent times, and at different rates. In addition, each of theprocessing pipelines 815 and 820 of some embodiments can be configureddifferently through different register values as described above byreference to the CIPU 400.

In some embodiments, a number of modules of the CIPU 400 include one ormore line/frame buffers for performing some or all of the module'soperations. For example, a filtering module 460 of some embodiments isimplemented to perform a 3×3 low-pass filtering. In such embodiments,the 3×3 low-pass filter processes three consecutive lines in the imagein order to apply the 3×3 low-pass filter on the middle line of thethree consecutive lines. Thus, the filtering module 460 of suchembodiments requires at least three line/frame buffers in order performthe 3×3 low-pass filtering. Other modules in the CIPU 400 also includeone or more line/frame buffers like the BPC module 420 and the LS module425, for example.

The processing pipelines of the CIPU 800 can each have differentline/frame buffer sizes in order to customize the image processing tocharacteristics of its corresponding camera. For instance, if one cameraof the dual camera mobile device has a 2048×1500 pixel sensor, theprocessing pipeline of the 2048×1500 pixel sensor can include line/framebuffers that are 2048 pixels wide. Similarly, if the other camera of thedual camera mobile device has a 640×480 pixel sensor, the processingpipeline of the 640×480 pixel sensor can include line/frame buffers thatare 640 pixels wide. That is, the size of the line/frame buffersincluded in the modules of one processing pipeline can be different fromthe size of the line/frame buffers included in the modules of anotherprocessing pipeline.

III. Video Conferencing

A. Video Conference Architecture

FIG. 9 conceptually illustrates a software architecture for a videoconferencing and processing module 900 of a dual camera mobile device ofsome embodiments. The video conferencing and processing module 900includes a CIPU driver 905, a media exchange module 910, and an encoderdriver 920 that are similar to the corresponding modules and drivers305, 310, and 320 described above by reference to FIG. 3 . The videoconferencing and processing module 900 also includes a video conferencemodule 925, a video conference client 945, and a network interface 950for performing a variety of video conferencing functions. Like the videoprocessing and encoding module 300, the video conferencing andprocessing module 900 processes and encodes images that are capturedfrom cameras of the dual camera mobile device.

As described above by reference to FIG. 3 , the media exchange module910 allows consumers and producers of media content in the device toexchange media content and instructions regarding the processing of themedia content, the CIPU driver 905 serves as a communication interfacewith the captured image processing unit (CIPU) 955, and the encoderdriver 920 serves as a communication interface with the encoder hardware960 (e.g., an encoder chip, an encoding component on a system on chip,etc.).

The video conference module 925 of some embodiments handles variousvideo conferencing functions such as image processing, video conferencemanagement, and networking. As shown, the video conference module 925interacts with the media exchange module 910, the video conferenceclient 945, and the network interface 950. In some embodiments, thevideo conference module 925 receives instructions from and sendsinstructions to the video conference client 945. The video conferencemodule 925 of some embodiments also sends data to and receives data fromnetworks (e.g., a local area network (LAN), a wireless local areanetwork (WLAN), a wide area network (WAN), a network of networks, a codedivision multiple access (CDMA) network, a GSM network, etc.) throughthe network interface 950.

The video conference module 925 includes an image processing layer 930,a management layer 935, and a network layer 940. In some embodiments,the image processing layer 930 performs image processing operations onimages for video conferencing. For example, the image processing layer930 of some embodiments performs exposure adjustment, image resizing,perspective correction, and dynamic range adjustment as described infurther detail below. The image processing layer 930 of some embodimentssends requests through the media exchange module 910 for images from theCIPU 955.

The management layer 935 of some embodiments controls the operation ofthe video conference module 925. For instance, in some embodiments, themanagement layer 935 initializes a camera/cameras of the dual cameramobile device, processes images and audio to transmit to a remotedevice, and processes images and audio received from the remote device.In some embodiments, the management layer 935 generates composite (e.g.,PIP) displays for the device. Moreover, the management layer 935 maychange the operation of the video conference module 925 based onnetworking reports received from the network layer 940.

In some embodiments, the network layer 940 performs some or all of thenetworking functionalities for video conferencing. For instance, thenetwork layer 940 of some embodiments establishes a network connection(not shown) between the dual camera mobile device and a remote device ofa video conference, transmits images to the remote device, and receivesimages from the remote device, among other functionalities, as describedbelow. In addition, the network layer 940 receives networking data suchas packet loss, one-way latency, and roundtrip delay time, among othertypes of data, processes such data, and reports the data to themanagement layer 935.

The video conference client 945 of some embodiments is an applicationthat may use the video conferencing functions of the video conferencemodule 925 such as a video conferencing application, a voice-over-IP(VOIP) application (e.g., Skype), or an instant messaging application.In some embodiments, the video conference client 945 is a stand-aloneapplication while in other embodiments the video conference client 945is integrated into another application.

In some embodiments, the network interface 950 is a communicationinterface that allows the video conference module 925 and the videoconference client 945 to send data and receive data over a network(e.g., a cellular network, a local area network, a wireless network, anetwork of networks, the Internet, etc.) through the network interface950. For instance, if the video conference module 925 wants to send data(e.g., images captured by cameras of the dual camera mobile device) toanother device on the Internet, the video conference module 925 sendsthe images to the other device through the network interface 950.

B. Video Conference Set Up

FIG. 10 conceptually illustrates an example video conference requestmessaging sequence 1000 of some embodiments. This figure shows the videoconference request messaging sequence 1000 among a video conferenceclient 1010 running on a device 1005, a video conference server 1015,and a video conference client 1025 running on a device 1020. In someembodiments, the video conference clients 1010 and 1025 are the same asthe video conference client 945 shown in FIG. 9 . As shown in FIG. 10 ,one device (i.e., the device 1005) requests a video conference andanother device (i.e., the device 1020) responds to such request. Thedual camera mobile device described in the present application canperform both operations (i.e., make a request and respond to a request).

The video conference server 1015 of some embodiments routes messagesamong video conference clients. While some embodiments implement thevideo conference server 1015 on one computing device, other embodimentsimplement the video conference server 1015 on multiple computingdevices. In some embodiments, the video conference server is a publiclyaccessible server that can handle and route messages for numerousconferences at once. Each of the video conference clients 1010 and 1025of some embodiments communicates with the video conference server 1015over a network (e.g., a cellular network, a local area network, awireless network, a network of networks, the Internet etc.) through anetwork interface such as the network interface 950 described above.

The video conference request messaging sequence 1000 of some embodimentsstarts when the video conference client 1010 receives (at operation 1) arequest from a user of the device 1005 to start a video conference withthe device 1020. The video conference client 1010 of some embodimentsreceives the request to start the video conference when the user of thedevice 1005 selects a user interface (UI) item of a user interfacedisplayed on the device 1005. Examples of such user interfaces areillustrated in FIG. 11 and FIG. 14 , which are described below.

After the video conference client 1010 receives the request, the videoconference client 1010 sends (at operation 2) a video conferencerequest, which indicates the device 1020 as the recipient based on inputfrom the user, to the video conference server 1015. The video conferenceserver 1015 forwards (at operation 3) the video conference request tothe video conference client 1025 of the device 1020. In someembodiments, the video conference server 1015 forwards the videoconference request to the video conference client 1025 using pushtechnology. That is, the video conference server 1015 initiates thetransmission of the video conference request to the video conferenceclient 1025 upon receipt from the video conference client 1010, ratherthan waiting for the client 1025 to send a request for any messages.

When the video conference client 1025 of some embodiments receives thevideo conference request, a user interface is displayed on the device1020 to indicate to the user of the device 1020 that the user of thedevice 1005 sent a request to start a video conference and to prompt theuser of the device 1020 to accept or reject the video conferencerequest. An example of such a user interface is illustrated in FIG. 12 ,which is described below. In some embodiments, when the video conferenceclient 1025 receives (at operation 4) a request to accept the videoconference request from the user of the device 1005, the videoconference client 1025 sends (at operation 5) a video conferenceacceptance to the video conference server 1015. The video conferenceclient 1025 of some embodiments receives the request to accept the videoconference request when the user of the device 1020 selects a userinterface item of a user interface as illustrated in FIG. 12 , forexample.

After the video conference server 1015 receives the video conferenceacceptance from the video conference client 1025, the video conferenceserver 1015 forwards (at operation 6) the video conference acceptance tothe video conference client 1010. Some embodiments of the videoconference server 1015 forward the video conference acceptance to thevideo conference client 1010 using the push technology described above.

Upon receiving the video conference acceptance, some embodimentsestablish (at operation 7) a video conference between the device 1005and the device 1020. Different embodiments establish the videoconference differently. For example, the video conference establishmentof some embodiments includes negotiating a connection between the device1005 and the device 1020, determining a bit rate at which to encodevideo, and exchanging video between the device 1005 and the device 1020.

In the above example, the user of the device 1020 accepts the videoconference request. In some embodiments, the device 1020 can beconfigured (e.g., through the preference settings of the device) toautomatically accept incoming video conference requests withoutdisplaying a UI. Moreover, the user of the device 1020 can also reject(at operation 4) the video conference request (e.g., by selecting a userinterface item of a user interface displayed on the device 1020).Instead of sending a video conference acceptance, the video conferenceclient 1025 sends a video conference rejection to the video conferenceserver 1015, which forwards the video conference rejection to the videoconference client 1010. The video conference is then never established.

1. Video Conference Setup User Interface

In some embodiments, a video conference is initiated based on an ongoingphone call. That is, while the user of a mobile device is engaged in aphone call with a second user, the user can turn the phone call into avideo conference with the permission of the other party. For someembodiments of the invention, FIG. 11 illustrates the start of such avideo conference by a dual camera handheld mobile device 1100. Thisfigure illustrates the start of the video conference in terms of fiveoperational stages 1110, 1115, 1120, 1125, and 1130 of a user interface(“UI”) 1105 of the device 1100.

As shown in FIG. 11 , the UI 1105 includes a name field 1135, aselection menu 1140, and a selectable UI item 1145. The name field 1135displays the name of the person on the other end of the phone call, withwhom a user would like to request a video conference. In this example,the selectable UI item 1145 (which can be implemented as a selectablebutton) provides a selectable End Call option for the user to end thephone call. The selection menu 1140 displays a menu of selectable UIitems, such as a Speakerphone item 1142, a Mute item 1144, a Keypad item1146, a Phonebook item 1148, a Hold item 1152, a Video Conference item1154, etc. Different embodiments display the selection menu differently.For the embodiments illustrated by FIG. 11 , the selection menu 1140includes several equally sized icons, each of which represents adifferent operation. Other embodiments provide a scrollable menu, orgive priority to particular items (e.g., by making the items larger).

The operation of the UI 1105 will now be described by reference to thestate of this UI during the five stages, 1110, 1115, 1120, 1125, and1130 that are illustrated in FIG. 11 . In the first stage 1110, a phonecall has been established between the handheld mobile device user andNancy Jones. The second stage 1115 displays the UI 1105 after the userselects the selectable Video Conference option 1154 (e.g., through asingle finger tap by finger 1150) to activate a video conference tool.In this example, the Video Conference option 1154 (which can beimplemented as a selectable icon) allows the user to start a videoconference during the phone call. In the second stage, the VideoConference option 1154 is highlighted to indicate that the videoconference tool has been activated. Different embodiments may indicatesuch a selection in different ways (e.g., by highlighting the border orthe text of the item).

The third stage 1120 displays the UI 1105 after the device 1100 hasstarted the video conference process with the selection of the VideoConference option 1154. The third stage is a transitional hold stagewhile the device waits for the video conference to be established (e.g.,while the device waits for the device on the other end of the call toaccept or reject the video conference). In the third stage 1120, theuser of the device 1100 can still talk to the user of the other device(i.e., Nancy Jones) while the video conference connection is beingestablished. In addition, some embodiments allow the user of the device1100 to cancel the video conference request in the third stage 1120 byselecting a selectable UI item displayed on the UI 1105 (not shown) forcanceling the video conference request. During this hold stage,different embodiments use different displays in the UI 1105 to indicatethe wait state.

As shown in FIG. 11 , in some embodiments the wait state of the thirdstage is illustrated in terms of a full screen display of a video beingcaptured by the device 1100 along with a “Preview” notation at thebottom of this video. Specifically, in FIG. 11 , the third stage 1120illustrates the start of the video conference process by displaying in adisplay area 1160 of the UI 1105 a full screen presentation of the videobeing captured by the device's camera. In some embodiments, the frontcamera is the default camera selected by the device at the start of avideo conference. Often, this front camera points to the user of thedevice at the start of the video conference. Accordingly, in the exampleillustrated in FIG. 11 , the third stage 1120 illustrates the device1100 as presenting a full screen video of the user of the device 1100.The wait state of the device is further highlighted by the “Preview”designation 1165 below the video appearing in the display area 1160during the third stage 1120.

The transitional third hold stage 1120 can be represented differently insome embodiments. For instance, some embodiments allow the user of thedevice 1100 to select the back camera as the camera for starting thevideo conference. To allow for this selection, some embodiments allowthe user to specify (e.g., through a menu preference setting) the backcamera as the default camera for the start of a video conference, and/orallow the user to select the back camera from a menu that displays theback and front cameras after the user selects the Video Conferenceoption 1154. In either of these situations, the UI 1105 (e.g., displayarea 1160) displays a video captured by the back camera during the thirdhold stage 1120.

Also, other embodiments might indicate the activation of the videoconference tool by displaying the smaller version of the video capturedby the device 1100, by displaying a still image that is stored on thedevice 1100, by providing a message to highlight the wait state of thedevice (e.g., by showing “Conference Being Established”), by notdisplaying the “Preview” designation, etc. Also, in the third stage1120, the UI 1105 of some embodiments provides an End button (not shown)to allow the user to cancel entering the video conference and revertback to the phone call if he decides not to enter the video conferenceat this stage (e.g., while the user is waiting for the remote user torespond to his request).

The fourth stage 1125 illustrates the UI 1105 in a transitional stateafter the remote user has accepted the video conference request and avideo conference connection has been established. In this transitionalstate, the display area 1160 that displays the video of the local user(that is being captured by the front camera in this example) graduallydecreases in size (i.e., gradually shrinks), as indicated by the arrows1175. The display area 1160 (i.e., the local user's video) shrinks sothat the UI 1105 can display a display area 1170 (e.g., a display window1170) that contains the video from a camera of the remote device behindthe display area 1160. In other words, the shrinking of the local user'svideo 1160 creates a PIP display 1180 that has a foreground insetdisplay 1160 of the local user's video and a background main display1170 of the remote user. In this example, the background main display1170 presents a video of a lady whose video is being captured by theremote device's front camera (e.g., Nancy Jones, the user of the remotedevice) or a lady whose video is being captured by the remote device'sback camera (e.g., a lady whose video is being captured by Nancy Jones).One of ordinary skill will realize that the transitional fourth stageshown in FIG. 11 is simply one exemplary approach used by someembodiments, and that other embodiments might animate the transitionalfourth stage differently.

The fourth stage 1125 also illustrates a selectable UI item 1132 in alower display area 1155. The selectable UI item 1132 (which can beimplemented as a selectable button) provides a selectable End Conferenceoption 1132 below the PIP display 1180. The user may select this EndConference option 1132 to end the video conference (e.g., through asingle finger tap). Different embodiments may allow the user to end theconference in different ways, such as by toggling a switch on the mobiledevice, by giving voice commands, etc. Moreover, different embodimentsmay allow the End Conference option 1132 to fade away during the videoconference, thereby allowing the PIP display 1180) to take up the entiredisplay area 1185. The End Conference option 1132 may then reappear at asingle finger tap at the bottom of the display area 1185, giving theuser access to the End Conference option 1132. In some embodiments, thelayout of the display area 1155 is same as the display area 1155described in further detail below.

The fifth stage 1130 illustrates the UI 1105 after the animation of thefourth transitional state 1125 has ended. Specifically, the fifth stage1130 illustrates a PIP display 1180 that is presented by the UI 1105during the video conference. As mentioned above, this PIP display 1180includes two video displays: a larger background display 1170 from theremote camera and a smaller foreground inset display 1160 from the localcamera.

This PIP display 1180 is only one manner of presenting a composite viewof the videos being captured by the remote and local devices. Inaddition to this composite view, the devices of some embodiments provideother composite views. For example, instead of having a largerbackground display 1170 of the remote user, the larger backgrounddisplay 1170 can be of the local user and the smaller foreground insetdisplay 1160 of the remote user. As further described below, someembodiments allow a user to switch during a video conference between thelocal cameras and/or remote cameras as the cameras for the inset andmain views in the PIP display 1180.

Also, some embodiments allow the local and remote videos to appear inthe UI 1105 in two side-by-side display areas (e.g., left and rightdisplay windows, or top and bottom display windows) or two diagonallyaligned display areas. The manner of the PIP display or a defaultdisplay mode may be specified by the user in some embodiments throughthe preference settings of the device or through controls that the usercan select during a video conference, as further described below.

When the user of the device 1100 of FIG. 11 invites the remote user to avideo conference, the remote user may accept or reject the invitation.FIG. 12 illustrates a UI 1205 of the remote user's device 1200 at sixdifferent stages 1210, 1215, 1220, 1225, 1230, and 1235 that show thesequence of operations for presenting and accepting a video conferenceinvitation at the remote user's device. The description of the UI 1205below refers to the user of the device 1200 (i.e., the device thatreceives the video conference request) as the invite recipient, and theuser of the device 1100 (i.e., the device that sends the videoconference request) as the invite requestor. Also, in this example, itis assumed that the invite recipient's device 1200 is a dual cameradevice, like that of the invite requestor. However, in other examples,one or both of these devices are single camera devices.

The first stage 1210 illustrates the UI 1205 when the invite recipientreceives an invitation to a video conference from the invite requestor,John Smith. As shown in FIG. 12 , the UI 1205 in this stage includes aname field 1235, a message field 1240, and two selectable UI items 1245and 1250. The name field 1235 displays the name of a person who isrequesting a video conference. In some embodiments, the name field 1235displays a phone number of the person who is requesting a videoconference instead of the name of the person. The message field 1240displays an invite from the invite requestor to the invite recipient. Inthis example, the “Video Conference Invitation” in the field 1240indicates that the invite requestor is requesting a video conferencewith the invite recipient. The selectable UI items 1245 and 1250 (whichcan be implemented as selectable buttons) provide selectable DenyRequest and Accept Request options 1245 and 1250 for the inviterecipient to use to reject or accept the invitation. Differentembodiments may display these options differently and/or display otheroptions.

Upon seeing the “Video Conference Invitation” notation displayed in themessage field 1240, the invite recipient may deny or accept the requestby selecting the Deny Request option 1245 or Accept Request option 1250in the UI, respectively. The second stage 1215 illustrates that in theexample shown in FIG. 12 , the user selects the Accept Request option1250. In this example, this selection is made by the user's fingertapping on the Accept Request option 1250, and this selection isindicated through the highlighting of this option 1250. Other techniquesare provided in some embodiments to select the Accept or Deny Requestoptions 1245 and 1250 (e.g., double-tapping, etc.) to indicate theselection (e.g., highlighting the border or text of the UI item).

The third stage 1220 displays the UI 1205 after the invite recipient hasagreed to join the video conference. In this stage, the UI 1205 entersinto a preview mode that shows a full screen presentation of the videofrom the remote device's front camera in a display area 1244. The frontcamera in this case is pointed to the user of the remote device (i.e.,Nancy Jones in this example). Accordingly, her image is shown in thispreview mode. This preview mode allows the invite recipient to make surethat her video is displayed properly and that she is happy with herappearance before the video conference begins (e.g., before actualtransmission of the video begins). In some embodiments, a notation, suchas a “Preview” notation, may be displayed below the display area 1244 toindicate that the invite recipient is in the preview mode.

Some embodiments allow the invite recipient to select the back camera asthe default camera for the start of the video conference, or to selectthe front or back camera at the beginning of the video conference, asfurther described below. Also, other embodiments display the previewdisplay of the invite recipient differently (e.g., in a smaller imageplaced in the corner of the display area 1244). Yet other embodiments donot include this preview mode, but rather start the video conferenceimmediately after the invite recipient accepts the request.

In the third stage, the UI 1205 shows two selectable UI items 1275 and1246, one of which overlaps the display area 1244 while the other isbelow this display area 1244. The selectable UI item 1275 is an Acceptbutton 1275 that the user may select to start video conferencing. Theselectable UI item 1246 is an End button 1246 that the invite recipientcan select if she decides not to join the video conference at thisstage.

The fourth stage 1225 displays the UI 1205 after the invite recipientselects the Accept button 1275. In this example, the Accept button 1275is highlighted to indicate that the invite recipient is ready to startthe video conference. Such a selection may be indicated in differentways in other embodiments.

The fifth stage 1230 illustrates the UI 1205 in a transitional stateafter the invite recipient has accepted the video conference request. Inthis transitional stage, the display area 1244 that displays the videoof the invite recipient (that is being captured by the front camera inthis example) gradually decreases in size (i.e., gradually shrinks), asindicated by the arrows 1260. The invite recipient's video shrinks sothat the UI 1205 can display a display area 1265 (e.g., a display window1265) that contains the video from a camera of the invite requestorbehind the display area 1244. In other words, the shrinking of theinvite recipient's video creates a PIP display 1280 that has aforeground inset display area 1244 of the invite recipient's video and abackground main display 1265 of the invite requestor.

In this example, the background main display 1265 presents a video of aman whose video is being captured by the local device's front camera(i.e., John Smith, the user of the local device 1100). In anotherexample, this video could have been that of a man whose video is beingcaptured by the local device's back camera (e.g., a man whose video isbeing captured by John Smith). Different embodiments may animate thistransitional fifth stage differently.

The UI at the fifth stage 1230 also displays a display area 1155 (e.g.,a tool bar or a menu bar) that includes selectable UI item 1285 (e.g.,mute button 1285) for muting the audio of the other user during thevideo conference, selectable UI item 1287 (e.g., end conference button1287) for ending the video conference, and selectable UI item 1289(e.g., switch camera button 1289) for switching cameras, which isdescribed in further detail below. As such, the invite recipient mayselect any of the selectable UI items 1285-1289 (e.g., through a singlefinger tap) to perform the desired operation during the videoconference. Different embodiments may allow the invite recipient toperform any of the operations in different ways, e.g., by toggling aswitch on the mobile device, by giving voice commands, etc.

Although FIG. 12 shows an example layout for the display area 1155, someembodiments provide different layouts of the display area 1155 such asthe layout of display area 1155 of FIG. 11 , which includes just aselectable End Conference UI item 1132 for ending the video conference.Other layouts of display area 1155 can include any number of differentselectable UI items for performing different functions. Moreover, thefifth stage 1230 shows the display area 1155 displayed at the bottom ofthe UI 1205. Different embodiments of the display area 1155 can bedisplayed at different locations within the UI 1205 and/or defined asdifferent shapes.

FIG. 12 shows the display area 1155 as a static display area (i.e., thedisplay area 1155 is always displayed). However, in some embodiments thedisplay area 1155 is a dynamic display area. In some such embodiments,the display area 1155 is not ordinarily displayed. Rather, the displayarea 1155 is displayed when a triggering event is received (e.g., a userselection such tapping the display area 1280 once, a voice command,etc.). The display area 1155 disappears after a user selection isreceived (e.g., selecting the selectable mute UI item 1285) or a definedamount of time (e.g., 3 seconds), which can be specified by the userthrough the preference settings of the mobile device or the videoconference application. In some such embodiments, the display area 1155is automatically displayed after the video conference starts anddisappears in the same manner mentioned above.

The sixth stage 1235 illustrates the UI 1205 after the animation of thefifth transitional stage has ended. Specifically, the sixth stageillustrates a PIP display 1280 that is presented by the UI 1205 duringthe video conference. As mentioned above, this PIP display 1280 includestwo video displays: a larger background display 1265 from the localcamera and a smaller foreground inset display 1244 from the remotecamera. This PIP display 1280 is only one manner of presenting acomposite view of the videos being captured by the remote and localdevices. In addition to this composite view, the devices of someembodiments provide other composite views. For example, instead ofhaving a larger background display of the invite recipient, the largerbackground display can be of the invite requestor and the smallerforeground inset display of the invite recipient. As further describedbelow, some embodiments allow a user to control the inset and main viewsin a PIP display to switchably display the local and remote cameras.Also, some embodiments allow the local and remote videos to appear inthe UI 1205 in two side-by-side display areas (e.g., left and rightdisplay windows, or top and bottom display windows) or two diagonallyaligned display areas. The manner of PIP display or a default displaymode may be specified by the user in some embodiments through thepreference settings of the device or through controls that the user canselect during a video conference, as further described below.

Although FIG. 12 shows the sequence of operations for presenting andaccepting a video conference invitation in terms of six differentoperational stages, some embodiments may implement the operation in lessstages. For instance, some of such embodiments may omit presenting thethird and fourth stages 1220 and 1225 and go from the second stage 1215to the fifth stage 1230 after the user selects the Accept Request option1250. Other embodiments that implement that operation (i.e., presentingand accepting a video conference invitation) in less stages may omit thefirst and second stages 1210 and 1215 and present the user with thethird stage 1220 when the invite recipient receives an invitation to avideo conference from the invite requestor.

FIG. 13 illustrates an example of performing the operation illustratedin FIG. 12 in less stages by combining the first and third stages intoone stage and the second and fourth stage into one stage. In particular,this figure illustrates a UI 1205 of the remote user's device 1200 atfive different stages 1390, 1392, 1394, 1230, and 1235. The first stage1390 is similar to the stage 1110 except the name field 1235 displaysthe name “John Smith” to indicate the name of the person on the otherend of the telephone call. That is, a phone call has been establishedbetween the user of the remote mobile device and the user of the localdevice (i.e., John Smith in this example). The second and third stages1392 and 1394 are similar to the first and second stages 1210 and 1215of FIG. 12 except the second and third stage 1392 and 1394 also show apreview of the user of the remote mobile device (i.e., Nancy Jones inthis example). The fourth and fifth stages 1230 and 1235 are the same asthe fifth and sixth stages 1230 and 1235 of FIG. 12 .

In addition to activating the video conference tool through a selectableoption during a phone call, some embodiments allow a user of a dualcamera device to initiate a video conference directly without having tomake a phone call first. FIG. 14 illustrates another such alternativemethod to initiate a video conference. This figure illustrates the UI1405 at seven different stages 1410, 1415, 1420, 1425, 1430, 1435, and1440 that show an alternative sequence of operations for starting avideo conference.

In the first stage 1410, a user is looking through a contacts list onthis mobile device for the person with whom he wants to engage in avideo conference, similar to how he would find a contact to call. In thesecond stage 1415, the user selects the person 1455 with whom he wouldlike to have a video conference (e.g., through a single finger tap 1460on the person's name 1455). This selection triggers the UI 1405 todisplay the contact's information and various user selectable options.In this example, Jason's name 1455 is highlighted to indicate that thisis the person with whom the user would like to have a video conference.Different embodiments may indicate such a selection in different ways.While the second stage 1415 allows the user of the device 1400 to selecta person with whom the user would like to have a video conferencethrough a contact list, some embodiments allow the user to select theperson through a “Recents” call history that lists a particular numberor name of a person with whom the user of the device 1400 recently had avideo conference or a phone call.

In the third stage 1420, the UI 1405 displays the selected person'sinformation 1462 and various selectable UI items 1468, 1472, and 1470after the person's name 1455 has been selected. In this example, one ofthe various selectable UI items 1472 (which can be implemented as aselectable icon or button) provides a video conference tool. The VideoConference option 1472 allows the user to invite the person identifiedby the contact 1466 to a video conference. Different embodiments displaythe information 1462 and selectable UI items 1468, 1472, and 1470differently (e.g., in a different arrangement).

The fourth stage 1425 shows the user selecting the Video Conferenceoption 1472 (e.g., through a single finger tap). In this example, theVideo Conference option 1472 is highlighted to indicate that the videoconference tool 1472 has been activated. Such selections may beindicated differently in different embodiments (e.g., by highlightingthe text or border of the selected icon).

The fifth, sixth and seventh stages 1430, 1435, and 1440 are similar tothe third, fourth and fifth stages 1120, 1125, and 1130 illustrated inFIG. 11 and may be understood by reference to the discussion of thosestages. In brief, the fifth stage 1430 illustrates a transitionalholding stage that waits for the remote user to respond to theinvitation to a video conference. The sixth stage 1435 illustrates thatafter the remote user has accepted the video conference request, thedisplay area 1480 (that displays the video of the local user) graduallydecreases in size so the UI 1405 can show a display area 1492 thatcontains the video from a camera of the remote user behind the displayarea 1480. In the seventh stage 1440, the PIP display 1447 is presentedby the UI 1405 during the video conference. In some embodiments, thelayout of display area 1155 in the sixth stage 1435 and the seventhstage 1440 is like the layout of the display area 1155 of FIG. 12 ,described above.

FIGS. 10, 11, 12, 13, and 14 show several ways of establishing a videoconference. In some embodiments, during a telephone call, audio data(e.g., voice) is transmitted through one communication channel (over acommunication network like a circuit-switched communication network or apacket-switched communication network) and, during a video conference,audio data is transmitted through another communication channel. Thus,in such embodiments, audio data (e.g., voice) is transmitted through acommunication channel before the video conference is established, andonce the video conference is established, audio is transmitted through adifferent communication channel (instead of the communication channelused during the telephone call).

In order to provide a seamless transition (e.g., handoff) of audio datafrom the telephone call to the video conference, some embodiments do notterminate the telephone call before establishing the video conference.For instance, some embodiments establish a peer-to-peer video conferenceconnection (e.g., after completing the message sequence illustrated inFIG. 10 ) before terminating the phone call and starting to transmitaudio/video data through the peer-to-peer communication session.Alternatively, other embodiments establish a peer-to-peer videoconference connection (e.g., after completing the message sequenceillustrated in FIG. 10 ) and start transmitting audio/video data throughthe peer-to-peer communication session, before terminating the phonecall and starting to present the received audio/video data.

A peer-to-peer video conference connection of some embodiments allowsthe mobile devices in the video conference to directly communicate witheach other (instead of communicating through a central server, forexample). Some embodiments of a peer-to-peer video conference allow themobile devices in the video conferences to share resources with eachother. For instance, through a control communication channel of a videoconference, one mobile device can remotely control operations of anothermobile device in the video conference by sending instructions from theone mobile device to the other mobile device to direct the other mobiledevice to process images differently (i.e., share its image processingresource) such as an exposure adjustment operation, a focus adjustmentoperation, and/or a switch camera operation, described in further detailbelow.

2. Dynamic Bit Rate Setup

Typically, mobile devices in a video conference communicate data (e.g.,audio and video images) to each other through communication channels ofdifferent types of communication networks such as different private andpublic wireless communication networks (e.g., cellular networks likeGSM, UMTS, etc.). Examples of such wireless communication networks willbe explained by reference to FIGS. 91 and 92 , below.

A communication network's available bandwidth for conducting a videoconference differ at different times due to the varying number of mobiledevices accessing the communication network at a given time. Theavailable bandwidth can even change during the video conference. Inaddition, flooding the communication network with high bit rates duringthe video conference or large amounts of signaling in an attempt tofigure out an optimal video conference bit rate is undesirable.

Because of these reasons, some embodiments employ a novel method forspecifying the initial optimal bit rate for the video conference. Toidentify the initial optimal bit rate for the video conference, thismethod starts a video conference at a particular bit rate andincrementally increases the bit rate at particular intervals if theseembodiments do not detect network conditions that would degrade thevideo conference quality.

One example of such embodiments is illustrated in FIG. 15 . This figureconceptually illustrates a process 1500 of some embodiments of theinvention for setting the bit rate for a video conference. The process1500 is performed as part of a video conference setup (e.g., as part ofthe video conference establishment illustrated in FIG. 10 ) todynamically determine a bit rate for transmitting data (e.g., audio andvideo images) based on various network conditions. In some embodiments,the process 1500 is performed by the management layer 935 of the videoconference module 925 described above by reference to FIG. 9 . A moredetailed version of this video conference module will be described belowby reference to FIG. 16 .

As shown in FIG. 15 , the process 1500 starts by setting (at 1505) a bitrate at an initial bit rate. In some embodiments, the initial bit rateis a default baseline rate for the device. Some embodiments, though,allow a user to specify the initial bit rate. At 1505, the process 1500also starts the video conference transmitting data (e.g., audio andvideo images) through one or more communication channels at the initialbit rate to a remote device.

Next, the process 1500 identifies (at 1510) a set of network conditionparameters received from the remote device in the video conference. Insome embodiments, the local device receives the set of network conditionparameters from the remote device through a real-time transport protocol(RTP) communication session established at the start of the videoconference. For instance, some embodiments provide the network conditionparameters through the extension feature of RTP. Moreover, the RTPextension feature of some embodiments can be used to transmit any typeof information (such as the set of network condition parameters) byindicating the presence of an extension header in an RTP packet headerand defining an extension header for the additional information.

In different embodiments, devices in the video conference relaydifferent sets of network condition/congestion parameters. In theembodiments described below, the set of network condition parametersinclude one-way latency and a bandwidth estimation bit rate. In otherembodiments, the set of network condition parameters includes packetloss data and roundtrip time (RTT) delay data. As such, differentembodiments can include any number of different network conditionparameters in the set of network condition parameters.

In some embodiments, the set of network condition parameters receivedfrom the remote device of the video conference is based on data (e.g.,audio and video) transmitted from the local mobile device (i.e., themobile device performing the process 1500) to the remote device duringthe video conference at the initial bit rate set at operation 1505. Forexample, in some embodiments, the remote device can determine one-waylatency by calculating the time it takes for audio packets to propagatethrough the network connection from the local mobile device to theremote device by using timestamps of the audio packets. Specifically, insome embodiments, each audio packet is time stamped. In the absence ofpacket delay, the remote devices should receive the audio packets at setintervals that are equal to the difference in the time stamps. However,when there is one-way latency delay, the remote device receives theaudio packets at intervals greater than the difference in the timestamps.

Also, in some embodiments, the remote device determines a bandwidthestimation bit rate by examining the time a video packet is received,the time a consecutive video packet is received, and the size of theconsecutive video packet. That is, the difference in time between thereceipt of the two consecutive video packets and the size of the secondvideo packet is used to estimate the available bandwidth of the networkconnection. Some embodiments determine a bandwidth estimation bit rateby examining multiple pairs of consecutive video packets. The aboveexamples use specific types of data (i.e., audio data for determiningone-way latency and video data for determining a bandwidth estimationbit rate). However, other types of data communicated through the networkconnection between the local mobile device and the remote device can beused as well in some embodiments.

After identifying (at 1510) the set of network conditions, the process1500 then determines (at 1515) whether the one-way latency hasdeteriorated past a defined threshold amount. In some embodiments, thethreshold amount is defined as a particular latency amount and theone-way latency is determined to deteriorate past the threshold amountwhen the difference between the current one-way latency and a previousone-way latency exceeds the particular latency amount. In otherembodiments, the threshold amount is defined as a particular rate ofchange of one-way latencies. As such, the one-way latency is determinedto deteriorate past the threshold amount when the rate of change of aset of one-way latencies (e.g., the current one-way latency and previousone-way latencies) exceeds the particular rate of change.

When the one-way latency is determined to deteriorate past the thresholdamount, the process 1500 ends. Otherwise, the process 1500 determines(at 1520) whether the current bit rate has reached the bandwidthestimation bit rate. In some embodiments, the bandwidth estimation bitrate indicates an amount of available bandwidth (e.g., 15 kilobits/sec(kbps)) for the network connection. When the process 1500 determinesthat the current bit rate exceeds the bandwidth estimation bit rate, theprocess 1500 ends. When the process 1500 determines that the current bitrate does not exceed the bandwidth estimation bit rate, the process 1500proceeds to operation 1525.

At 1525, the process 1500 determines whether the current bit rate hasreached a defined maximum bit rate. When the process 1500 determinesthat the current bit rate exceeds the defined maximum bit rate, theprocess 1500 ends. Otherwise, the process 1500 proceeds to operation1530 to increase the current bit rate by a defined amount. Differentembodiments define the amount to increase the bit rate differently.Examples of a defined amount to increase the current bit rate include 32kbps, 64 kbps, among any number of other amounts to increase the bitrate.

Next, the process determines (at 1535) whether a defined amount of timehas elapsed. The defined amount of time can be 1 second, 2 seconds, 5seconds, or any other possible amounts of time since differentembodiments define the amount of time differently. The process 1500waits for the defined amount of time to elapse in order for the remotedevice to receive data (e.g., audio and video images) transmitted fromthe local mobile device at the newly increased bit rate (at operation1530) and determine network condition parameters based on the newlyincreased bit rate. If the process 1500 determines that the definedamount of time has not elapsed, the process 1500 returns to operation1535 until the defined amount of time has elapsed. When the process 1500determines that the defined amount of time has elapsed, the process 1500returns to operation 1510. The operation of the process 1500 from 1510onwards proceeds as described above until the process 1500 ends.

When the process 1500 ends (i.e., after operation 1515, 1520, or 1525),the setup of a bit rate for the video conference is complete and anoptimal bit rate has been determined. Since the available bandwidth forthe video conference can change during the video conference, someembodiments continue to adjust the bit rate based on the set of networkcondition parameters (i.e., the one-way latency and the bandwidthestimation bit rate) that are received from the remote device. The bitrate can be adjusted during the video conference by increasing the bitrate. For instance, if the process 1500 ended because the one-waylatency deteriorated past the defined threshold amount and, during thevideo conference, the one-way latency improves, some embodimentsincrease the bit rate. Similarly, if the process 1500 ended because thebit rate exceeded the bandwidth estimation bit rate and, during thevideo conference, the bandwidth estimation bit rate increases, someembodiments increase the bit rate.

In contrast, the bit rate can be adjusted during the video conference bydecreasing the bit rate. For example, if the one-way latency continuesto deteriorate past the defined threshold amount during the videoconference, some embodiments decrease the bit rate. Also, if the bitrate continues to exceed the bandwidth estimation bit rate (e.g., thebandwidth estimation bit rate continues to decrease) during the videoconference, some embodiments decrease the bit rate.

Moreover, the description of the process 1500 uses one-way latency and abandwidth estimation bit rate to determine whether to increase the bitrate. However, one of ordinary skill will realize that any number ofnetwork condition parameters can be used to determine whether toincrease the bit rate in different embodiments. For instance,determining whether to increase the bit rate can be based on solely RTTdelay data or packet loss data.

C. Video Conference Architecture

As mentioned above, FIG. 16 conceptually illustrates a softwarearchitecture for a video conferencing and processing module 1600 of adual camera mobile device of some embodiments. As shown, the videoconferencing and processing module 1600 includes a client application1665, a video conference module 1602, a media exchange module 1620, abuffer 1625, a captured image processing unit (CIPU) driver 1630, anencoder driver 1635, and a decoder driver 1640. In some embodiments, thebuffer 1625 is a frame buffer that stores images of a video for displayon a display 1645 of the dual camera mobile device.

In some embodiments, the client application 1665 is the same as thevideo conference client 945 of FIG. 9 . As mentioned above, the clientapplication 1665 may be integrated into another application orimplemented as a stand-alone application. The client application 1665may be an application that uses the video conferencing functions of thevideo conference module 1602, such as a video conferencing application,a voice-over-IP (VOIP) application (e.g., Skype), or an instantmessaging application.

The client application 1665 of some embodiments sends instructions tothe video conference module 1602 such as instructions to start aconference and end a conference, receives instructions from the videoconference module 1602, routes instructions from a user of the dualcamera mobile device to the video conference module 1602, and generatesuser interfaces that are displayed on the dual camera mobile device andallow a user to interact with the application.

D. Video Conference Manager

As shown in FIG. 16 , the video conference module 1602 includes a videoconference manager 1604, an image processing manager 1608, a networkingmanager 1614, and buffers 1606, 1610, 1612, 1616, and 1618. In someembodiments, the video conference module 1602 is the same as the videoconference module 925 illustrated in FIG. 9 and thus performs some orall of the same functions described above for the video conferencemodule 925.

In some embodiments, the video conference manager 1604 is responsiblefor initializing some or all of the other modules of the videoconference module 1602 (e.g., the image processing manager 1608 and thenetworking manager 1614) when a video conference is starting,controlling the operation of the video conference module 1602 during thevideo conference, and ceasing the operation of some or all of the othermodules of the video conference module 1602 when the video conference isending.

The video conference manager 1604 of some embodiments also processesimages received from one or more devices in the video conference andimages captured by one of both cameras of the dual camera mobile devicefor display on the dual camera mobile device. For instance, the videoconference manager 1604 of some embodiments retrieves decoded images,that were received from another device participating in the videoconference, from the buffer 1618 and retrieves images processed by CIPU1650 (i.e., images captured by the dual camera mobile device) from thebuffer 1606. In some embodiments, the video conference manager 1604 alsoscales and composites the images before displaying the images on thedual camera mobile device. That is, the video conference manager 1604generates the PIP or other composite views to display on the mobiledevice in some embodiments. Some embodiments scale the images retrievedfrom the buffers 1606 and 1618 while other embodiments just scale imagesretrieved from one of the buffers 1606 and 1618.

Although FIG. 16 illustrates the video conference manager 1604 as partof the video conference module 1602, some embodiments of the videoconference manager 1604 are implemented as a component separate from thevideo conference module 1602. As such, a single video conference manager1604 can be used to manage and control several video conference modules1602. For instance, some embodiments will run a separate videoconference module on the local device to interact with each party in amulti-party conference, and each of these video conference modules onthe local device are managed and controlled by the one video conferencemanager.

The image processing manager 1608 of some embodiments processes imagescaptured by the cameras of the dual camera mobile device before theimages are encoded by the encoder 1655. For example, some embodiments ofthe image processing manager 1608 perform one or more of exposureadjustment, focus adjustment, perspective correction, dynamic rangeadjustment, and image resizing on images processed by the CIPU 1650. Insome embodiments, the image processing manager 1608 controls the framerate of encoded images that are transmitted to the other device in thevideo conference.

Some embodiments of the networking manager 1614 manage one or moreconnections between the dual camera mobile device and the other deviceparticipating in the video conference. For example, the networkingmanager 1614 of some embodiments establishes the connections between thedual camera mobile device and the other device of the video conferenceat the start of the video conference and tears down these connections atthe end of the video conference.

During the video conference, the networking manager 1614 transmitsimages encoded by the encoder 1655 to the other device of the videoconference and routes images received from the other device of the videoconference to decoder 1660 for decoding. In some embodiments, thenetworking manager 1614, rather than the image processing manager 1608,controls the frame rate of the images that are transmitted to the otherdevice of the video conference. For example, some such embodiments ofthe networking manager 1614 control the frame rate by dropping (i.e.,not transmitting) some of the encoded frames that are supposed to betransmitted to the other device of the video conference.

As shown, the media exchange module 1620 of some embodiments includes acamera source module 1622, a video compressor module 1624, and a videodecompressor module 1626.

The media exchange module 1620 is the same as the media exchange module310 shown in FIG. 3 , with more detail provided. The camera sourcemodule 1622 routes messages and media content between the videoconference module 1602 and the CIPU 1650 through the CIPU driver 1630,the video compressor module 1624 routes message and media contentbetween the video conference module 1602 and the encoder 1655 throughthe encoder driver 1635, and the video decompressor module 1626 routesmessages and media content between the video conference module 1602 andthe decoder 1660 through the decoder driver 1640. Some embodimentsimplement the TNR module 315 included in the media exchange module 310(not shown in FIG. 16 ) as part of the camera source module 1622 whileother embodiments implement the TNR module 315 as part of the videocompressor module 1624.

In some embodiments, the CIPU driver 1630 and the encoder driver 1635are the same as the CIPU driver 305 and the encoder driver 320illustrated in FIG. 3 . The decoder driver 1640 of some embodiments actsas a communication interface between the video decompressor module 1626and decoder 1660. In such embodiments, the decoder 1660 decodes imagesreceived from the other device of the video conference through thenetworking manager 1614 and routed through the video decompressor module1626. After the images are decoded, they are sent back to the videoconference module 1602 through the decoder driver 1640 and the videodecompressor module 1626.

In addition to performing video processing during a video conference,the video conferencing and processing module 1600 for the dual cameramobile device of some embodiments also performs audio processingoperations during the video conference. FIG. 17 illustrates such asoftware architecture. As shown, the video conferencing and processingmodule 1600 includes the video conference module 1602 (which includesthe video conference manager 1604, the image processing manager 1608,and the networking manager 1614), the media exchange module 1620, andthe client application 1665. Other components and modules of the videoconferencing and processing module 1600 shown in FIG. 16 are omitted inFIG. 17 to simplify the description. The video conferencing andprocessing module 1600 also includes frame buffers 1705 and 1710, audioprocessing manager 1715, and audio driver 1720. In some embodiments, theaudio processing manager 1715 is implemented as a separate softwaremodule while in other embodiments the audio processing manager 1715 isimplemented as part of the media exchange module 1620.

The audio processing manager 1715 processes audio data captured by thedual camera mobile device for transmission to the other device in thevideo conference. For example, the audio processing manager 1715receives audio data through the audio driver 1720, which is captured bymicrophone 1725, and encodes the audio data before storing the encodedaudio data in the buffer 1705 for transmission to the other device. Theaudio processing manager 1715 also processes audio data captured by andreceived from the other device in the video conference. For instance,the audio processing manager 1715 retrieves audio data from the buffer1710 and decodes the audio data, which is then output through the audiodriver 1720 to the speaker 1730.

In some embodiments, the video conference module 1602 along with theaudio processing manager 1715 and its associated buffers are part of alarger conference module. When a multi-participant audio conference isconducted between several devices without exchange of video content,this video conferencing and processing module 1600 only uses thenetworking manager 1614 and the audio processing manager 1715 tofacilitate the exchange of audio over an Internet Protocol (IP) layer.

The operation of the video conference manager 1604 of some embodimentswill now be described by reference to FIG. 18 . FIG. 18 conceptuallyillustrates a process 1800 performed by a video conference manager ofsome embodiments such as video conference manager 1604 illustrated inFIG. 16 . This can be equivalent to being performed by the managementlayer 935 of FIG. 9 . In some embodiments, the video conference manager1604 performs process 1800 when a user of the dual camera mobile deviceaccepts (e.g., through a user interface displayed on the dual cameramobile device) a video conference request or when a user of anotherdevice accepts a request sent by the user of the dual camera mobiledevice.

The process 1800 begins by receiving (at 1805) instructions to start avideo conference. In some embodiments, the instructions are receivedfrom the client application 1665 or are received from a user through auser interface displayed on the dual camera mobile device and forwardedto the video conference manager 1604 by the client application 1665. Forexample, in some embodiments, when a user of the dual camera mobiledevice accepts a video conference request, the instructions are receivedthrough the user interface and forwarded by the client application. Onthe other hand, when a user of the other device accepts a request sentfrom the local device, some embodiments receive the instructions fromthe client application without user interface interaction (althoughthere may have been previous user interface interaction to send out theinitial request).

Next, the process 1800 initializes (at 1810) a first module thatinteracts with the video conference manager 1604. The modules of someembodiments that interact with the video conference manager 1604 includethe CIPU 1650, the image processing manager 1608, the audio processingmanager 1715, and the networking manager 1614.

In some embodiments, initializing the CIPU 1650 includes instructing theCIPU 1650 to start processing images captured by one or both cameras ofthe dual camera mobile device. Some embodiments initialize the imageprocessing manager 1608 by instructing the image processing manager 1608to start retrieving images from the buffer 1610 and processing andencoding the retrieved images. To initialize the audio processingmanager 1715, some embodiments instruct the audio processing manager1715 to begin encoding audio data captured by the microphone 1725 anddecoding audio data stored in the buffer 1710 (which was received fromthe other device) in order to output to the speaker 1730. Theinitializing of the networking manager 1614 of some embodiments includesinstructing the networking manager 1614 to establish a networkconnection with the other device in the video conference.

The process 1800 then determines (at 1815) whether there are any modulesleft to initialize. When there are modules left to initialize, theprocess 1800 returns to operation 1810 to initialize another of themodules. When all of the required modules have been initialized, theprocess 1800 generates (at 1820) composite images for displaying on thedual camera mobile device (i.e., local display). These composite imagesmay include those shown in FIG. 65 , described below (i.e., PIP or othercomposite displays), and can include various combinations of images fromthe cameras of the local dual camera mobile device and images fromcameras of the other device participating in the video conference.

Next, the process 1800 determines (at 1825) whether a change has beenmade to the video conference. Some embodiments receive changes to thevideo conference through user interactions with a user interfacedisplayed on the dual camera mobile device while other embodimentsreceive changes to the video conference from the other device throughthe networking manager 1614 (i.e., remote control). The changes to videoconference settings may also be received from the client application1665 or other modules in the video conference module 1602 in someembodiments. The video conference settings may also change due tochanges in the network conditions.

When a change has been made, the process 1800 determines (at 1830)whether the change to the video conference is a change to a networksetting. In some embodiments, the changes are either network settingchanges or image capture setting changes. When the change to the videoconference is a change to a network setting, the process modifies (at1840) the network setting and then proceeds to operation 1845. Networksetting changes of some embodiments include changing the bit rate atwhich images are encoded or the frame rate at which the images aretransmitted to the other device.

When the change to the video conference is not a change to a networksetting, the process 1800 determines that the change is a change to animage capture setting and then proceeds to operation 1835. The process1800 then performs (at 1835) the change to the image capture setting. Insome embodiments, change to the image capture settings may includeswitching cameras (i.e., switching which camera on the dual cameramobile device will capture video), focus adjustment, exposureadjustment, displaying or not displaying images from one or both camerasof the dual camera mobile device, and zooming in or out of imagesdisplayed on the dual camera mobile device, among other setting changes.

At operation 1845, the process 1800 determines whether to end the videoconference. When the process 1800 determines to not end the videoconference, the process 1800 returns to operation 1820. When the process1800 determines that the video conference will end, the process 1800ends. Some embodiments of the process 1800 determine to end the videoconference when the process 1800 receives instructions from the clientapplication 1665 to end the video conference (i.e., due to instructionsreceived through the user interface of the local dual camera mobiledevice or received from the other device participating in the videoconference).

In some embodiments, the video conference manager 1604 performs variousoperations when the video conference ends that are not shown in process1800. Some embodiments instruct the CIPU 1650 to stop producing images,the networking manager 1614 to tear down the network connection with theother device in the video conference, and the image processing manager1608 to stop processing and encoding images.

E. Temporal Noise Reduction

Some embodiments include a specific temporal noise reduction module forprocessing video images to reduce noise in the video. The temporal noisereduction module of some embodiments compares subsequent images in avideo sequence to identify and eliminate unwanted noise from the video.

FIG. 19 conceptually illustrates a software architecture for such atemporal noise reduction (TNR) module 1900 of some embodiments. Someembodiments implement the TNR module 1900 as part of an application(e.g., as part of the media exchange module as shown in FIG. 3 ) whileother embodiments implement the TNR module 1900 as a stand-aloneapplication that is used by other applications. Yet other embodimentsimplement the TNR module 1900 as part of an operating system running onthe dual camera mobile device. In some embodiments, the TNR module 1900is implemented by a set of APIs that provide some or all of thefunctionalities of the TNR module 1900 to other applications.

As shown in FIG. 19 , the TNR module 1900 includes a TNR manager 1905, adifference module 1910, a pixel averaging module 1915, and a motionhistory module 1920. While FIG. 19 shows the three modules 1910, 1915,and 1920 as separate modules, some embodiments implement thefunctionalities of these modules, described below, in a single module.The TNR module 1900 of some embodiments receives as input an inputimage, a reference image, and a motion history. In some embodiments, theinput image is the image presently being processed while the referenceimage is the previous image in the video sequence to which the inputimage is compared. The TNR module 1900 outputs an output image (aversion of the input image with reduced noise) and an output motionhistory.

The TNR manager 1905 of some embodiments directs the flow of datathrough the TNR module 1900. As shown, the TNR manager 1905 receives theinput image, the reference image, and the motion history. The TNRmanager 1905 also outputs the output image and the output motionhistory. The TNR manager 1905 sends the input image and the referenceimage to the difference module 1910 and receives a difference image fromthe difference module 1910.

In some embodiments, the difference module 1910 processes the datareceived from the TNR manager 1905 and sends the processed data to theTNR manager 1905. As shown, the difference module 1910 receives theinput image and the reference image from the TNR manager 1905. Thedifference module 1910 of some embodiments generates a difference imageby subtracting the pixel values of one image from the pixel values ofthe other image. The difference image is sent to the TNR manager 1905.The difference image of some embodiments indicates the differencebetween the two images in order to identify sections of the input imagethat have changed and sections of the input image that have stayed thesame as compared to the previous image.

The TNR manager 1905 also sends the input image and reference image tothe pixel averaging module 1915. As shown, some embodiments also sendthe motion history to the pixel averaging module 1915 as well. Otherembodiments, however, might send only the input image and the referenceimage without the motion history. In either embodiments, the TNR manager1905 receives a processed image from the pixel averaging module 1915.

The pixel averaging module 1915 of some embodiments uses the motionhistory to determine whether to take an average of the pixels from theinput and reference images for a particular location in the image. Insome embodiments, the motion history includes a probability value foreach pixel in the input image. A particular probability value representsthe probability that the corresponding pixel in the input image haschanged (i.e., a dynamic pixel) with respect to the corresponding pixelin the reference image. For instance, if the probability value of aparticular pixel in the input image is 20, that indicates a probabilityof 20% that the particular pixel in the input image has changed withrespect to the corresponding pixel in the reference image. As anotherexample, if the probability value of a particular pixel in the inputimage is 0, that indicates that the particular pixel in the input imagehas not changed (i.e., a static pixel) with respect to the correspondingpixel in the reference image.

Different embodiments store the probability values of the input imagedifferently. Some embodiments might store the probability values of eachpixel of the input image in one array of data. Other embodiments mightstore the probability values in a matrix (e.g., an array of arrays) withthe same dimensions as the resolution of the images of the video. Forexample, if the resolution of the images of the video is 320×240, thenthe matrix is also 320×240.

When the pixel averaging module 1915 receives the motion history inaddition to the input image and reference image from the TNR manager1905, the pixel averaging module 1915 reads the probability values ofeach pixel in the input image. If the probability value for a particularpixel in the input image is below a defined threshold (e.g., 5%, 20%),the pixel averaging module 1915 averages the particular pixel value withthe corresponding pixel value in the reference image based on thepremise that there is not likely to be motion at the particular pixel,and thus differences between the images at that pixel may beattributable to noise.

If the probability for the particular pixel in the input image is notbelow the defined threshold, the pixel averaging module 1915 does notmodify the particular pixel of the input image (i.e., the pixel value atthat pixel stays the same as in the input image). This is because motionis more likely at the particular pixel, so differences between theimages are more likely to not be the result of noise. In someembodiments, when the motion history is not sent to the pixel averagingmodule 1915, the pixel averaging module 1915 averages each pixel in theinput image with the corresponding pixel in the reference image. Theprocessed image that is output by the pixel averaging module 1915 andsent to the TNR manager 1905 includes the input image pixel values forany pixels that were not averaged and the averaged pixel values for anypixels that were averaged by the pixel averaging module 1915.

In some embodiments, the motion history module 1920 processes datareceived from the TNR manager 1905 and sends the result data back to theTNR manager 1905. The motion history module 1920 of some embodimentsreceives the input image and the motion history from the TNR manager1905. Some embodiments input this data into a Bayes estimator in orderto generate a new motion history (i.e., a set of probability values)that can be used in the pixel averaging for the next input image. Otherembodiments use other estimators to generate the new motion history.

The operation of the TNR module 1900 will now be described by referenceto FIG. 20 . This figure conceptually illustrates a process 2000 of someembodiments for reducing temporal noise of images of a video. Theprocess 2000 starts by the TNR manager 1905 receiving (at 2005) an inputimage, a reference image, and a motion history. The input image is theimage presently being processed for noise reduction. In someembodiments, the reference image is the previous image of a sequence ofimages of the video as received from the CIPU. In other embodiments,however, the reference image is the output image generated from theprocessing of the previous input image (i.e., the output of TNR module1900). The motion history is the output motion history generated fromthe processing of the previous input image.

When the input image is a first image of the video, the TNR module 1900of some embodiments does not process (i.e., apply TNR to) the firstimage. In other words, the TNR manager 1905 receives the first image andjust outputs the first image. In other embodiments, when the input imageis the first image of the video, the first image is used as the inputimage and the reference image and the TNR module 1900 processes theimage as described below. Further, when the input image is the firstimage of the video, the motion history is empty (e.g., null, full ofzeros, etc.) and the TNR manager 1905 just outputs an empty motionhistory as the output motion history.

The TNR manager 1905 then determines (at 2010) whether the input imageis static. In order to make this determination, some embodiments sendthe input image and the reference image to the difference module 1910and receive a difference image from the difference module 1910. When thedifference between the two images is below a defined threshold (e.g., 5%difference, 10% difference, etc.), some embodiments classify the inputimage as static.

When the input image is a static image, the TNR manager 1905 sends theinput image and the reference image to the pixel averaging module 1915to average (at 2015) the pixels of the input image with the pixels ofthe reference image in order to reduce any noise from the static image.The process then proceeds to 2040, which is described below.

When the input image is not a static image, the TNR manager sends theinput image, reference image, and motion history to the pixel averagingmodule 1915 for processing. The pixel averaging module 1915 selects (at2020) a pixel in the input image. Using the motion history, the pixelaveraging module 1915 determines (at 2025) whether the pixels'probability of motion is below a particular threshold, as describedabove.

If the selected pixel's probability is below the particular threshold,the pixel averaging module 1915 averages (at 2030) the pixel of theinput image with the corresponding pixel in the reference image.Otherwise, the pixel is not averaged and the output image will be thesame as the input image at that particular pixel. The pixel averagingmodule 1915 then determines (at 2035) whether there are any unselectedpixels left in the input image. If any pixels have not yet beenprocessed, the process returns to operation 2020 to select the nextpixel. The pixel averaging module 1915 performs the operations 2020-2030until all pixels have been evaluated.

The process then updates (at 2040) the motion history. As shown in FIG.19 and described above, the motion history module 1920 updates themotion history based on the input image. The new motion history isoutput by the TNR manager along with the processed image from the pixelaveraging module.

F. Image Processing Manager & Encoder

In addition to temporal noise reduction and image processing operationsperformed by the CIPU and/or CIPU driver, some embodiments perform avariety of image processing operations at the image processing layer 930of the video conference module 925. These image processing operationsmay include exposure adjustment, focus adjustment, perspectivecorrection, adjustment of dynamic range, and image resizing, amongothers.

FIG. 21 conceptually illustrates a process 2100 for performing suchimage processing operations. In some embodiments, some or all of theoperations of the process 2100 are performed by a combination of theimage processing manager 1608 and the encoder driver 1635 of FIG. 16 .In some of such embodiments, the image processing manager 1608 performsthe pixel-based processing (e.g., resizing, dynamic range adjustment,perspective correction, etc.). Some embodiments perform process 2100during a video conference on images that are to be transmitted toanother device participating in the video conference.

The process 2100 will now be described by reference to FIG. 16 . Theprocess starts by retrieving (at 2105) an image from the buffer 1606. Insome embodiments, the retrieved image is an image of a video (i.e., animage in a sequence of images). This video may have been captured by acamera of a device on which the process 2100 is performed.

Next, the process 2100 performs (at 2110) exposure adjustment on theretrieved image. Some embodiments perform exposure adjustments through auser interface that is displayed on the dual camera mobile device. FIG.22 illustrates an example exposure adjustment operation of suchembodiments.

This figure illustrates the exposure adjustment operation by referenceto three stages 2210, 2215, and 2220 of a UI 2205 of a device 2200. Thefirst stage 2210 illustrates the UI 2205, which includes a display area2225 and a display area 1155. As shown, the display area 2225 displaysan image 2230 of a sun and a man with a dark face and body. The darkface and body indicates that the man is not properly exposed. The image2230 could be a video image captured by a camera of the device 2200. Asshown, the display area 1155 includes a selectable UI item 2250 forending the video conference. In some embodiments, the layout of thedisplay area 1155 is the same as the layout of the display area 1155 ofFIG. 12 , described above.

The second stage 2215 illustrates a user of the device 2200 initiatingan exposure adjustment operation by selecting an area of the displayarea 2225. In this example, a selection is made by placing a finger 2235anywhere within the display area 2225. In some embodiments, a userselects exposure adjustment from a menu of possible image settingadjustments.

The third stage 2220 shows an image 2240 of the man after the exposureadjustment operation is completed. As shown, the image 2240 is similarto the image 2230, but the man in the image 2240 is properly exposed. Insome embodiments, the properly exposed image is an image that iscaptured after the improperly exposed image. The exposure adjustmentoperation initiated in the second stage 2215 adjusts the exposure ofsubsequent images captured by the camera of the device 2200.

Returning to FIG. 21 , the process 2100 next performs (at 2115) focusadjustment on the image. Some embodiments perform focus adjustmentthrough a user interface that is displayed on the dual camera mobiledevice. FIG. 23 conceptually illustrates an example of such focusadjustment operations.

FIG. 23 illustrates a focus adjustment operation by reference to threedifferent stages 2310, 2315, and 2320 of a UI 2305 of a device 2300. Thefirst stage 2310 illustrates the UI 2305 including a display area 2325and a display area 1155. The display area 2325 presents a blurry image2330 of a man captured by a camera of the device 2300. The blurrinessindicates that the image 2330 of the man is out of focus. That is, thelens of the camera was not focused on the man when the image 2330 of theman was captured by the camera. Also, the image 2330 could be a videoimage captured by a camera of the device 2300. As shown, the displayarea 1155 includes a selectable UI item 2350 for ending the videoconference. In some embodiments, the layout of the display area 1155 isthe same as the layout of the display area 1155 of FIG. 12 , describedabove.

The second stage 2315 illustrates a user of the device 2300 initiating afocus adjustment operation by selecting an area of the display area2325. In this example, a selection is made by placing a finger 2335anywhere within the display area 2325. In some embodiments, a userselects focus adjustment from a menu of possible image settingadjustments.

The third stage 2320 shows an image 2340 of the man after the focusadjustment operation is completed. As shown, the image 2340 is the sameas the image 2330, but the man in the image 2340 appears sharper. Thisindicates that the lens of the camera is properly focused on the man. Insome embodiments, the properly focused image is an image that iscaptured after the improperly focused image. The focus adjustmentoperation initiated in the second stage 2315 adjusts the focus ofsubsequent images captured by the camera of the device 2300.

Back to FIG. 21 , the process 2100 performs (at 2120) image resizing onthe image. Some embodiments perform image resizing on the image toreduce the number of bits used to encode the image (i.e., lower the bitrate). In some embodiments, the process 2100 performs image resizing asdescribed below by reference to FIG. 26 .

The process 2100 next performs (at 2125) perspective correction on theimage. In some embodiments, the process 2100 performs perspectivecorrection as described in FIG. 24 below. Such perspective correctioninvolves using data taken by one or more accelerometer and/or gyroscopesensors that identifies orientation and movement of the dual cameramobile device. This data is then used to modify the image to correct forthe perspective being off.

After perspective correction is performed on the image, the process 2100adjusts (at 2130) the dynamic range of the image. In some embodiments,the dynamic range of an image is the range of possible values that eachpixel in the image can have. For example, an image with a dynamic rangeof 0-255 can be adjusted to a range of 0-128 or any other range ofvalues. Adjusting the dynamic range of an image can reduce the amount ofbits that will be used to encode the image (i.e., lower the bit rate)and thereby smooth out the image.

Adjusting the dynamic range of an image can also be used for variousother purposes. One purpose is to reduce image noise (e.g., the imagewas captured by a noisy camera sensor). To reduce noise, the dynamicrange of the image can be adjusted so that the black levels areredefined to include lighter blacks (i.e., crush blacks). In thismanner, the noise of the image is reduced. Another purpose of dynamicrange adjustment is to adjust one or more colors or range of colors inorder to enhance the image. For instance, some embodiments may assumethat the image captured by the front camera is an image of a person'sface. Accordingly, the dynamic range of the image can be adjusted toincrease the red and pink colors to make the person's cheeks appearrosy/rosier. The dynamic range adjustment operation can be used forother purposes as well.

Finally, the process 2100 determines (at 2135) one or more ratecontroller parameters that are used to encode the image. Such ratecontroller parameters may include a quantization parameter and a frametype (e.g., predictive, bi-directional, intra-coded) in someembodiments. The process then ends.

While the various operations of process 2100 are illustrated as beingperformed in a specific order, one of ordinary skill will recognize thatmany of these operations (exposure adjustment, focus adjustment,perspective correction, etc.) can be performed in any order and are notdependent on one another. That is, the process of some embodiments couldperform focus adjustment before exposure adjustment, or similarmodifications to the process illustrated in FIG. 21 .

1. Perspective Correction

As mentioned above, some embodiments perform perspective correction onan image before displaying or transmitting the image. In some cases, oneor more of the cameras on a dual camera mobile device will not beoriented properly with its subject and the subject will appear distortedin an uncorrected image. Perspective correction may be used to processthe images so that the images will closely reflect how the objects inthe images appear in person.

FIG. 24 conceptually illustrates a perspective correction process 2400performed by an image processing manager of some embodiments such asthat illustrated in FIG. 16 . The process 2400 of some embodiments isperformed by the image processing layer 930 shown in FIG. 9 (which maycontain an image processing manager 1608). Some embodiments perform theprocess 2400 at operation 2125 of process 2100, in order to correct theperspective of recently captured video images before displaying ortransmitting the images.

The process 2400 starts by receiving (at 2405) data from anaccelerometer sensor, which is a part of the dual camera mobile devicein some embodiments. The accelerometer sensor of some embodimentsmeasures the rate of change of the velocity of the device (i.e., thedevice's acceleration) along one or more axes. The process also receives(at 2410) data from a gyroscope sensor, which may also be a part of thedual camera mobile device in some embodiments. The gyroscope andaccelerometer sensors of some embodiments can be used individually or incombination to identify the orientation of the dual camera mobiledevice.

Next, the process 2400 determines (at 2415) the amount of perspectivecorrection to perform based on the data obtained from the accelerometerand gyroscope sensors. Generally, when the orientation is further offaxis, more perspective correction will be required to produce an optimalimage. Some embodiments calculate a warp parameter to represent theamount of perspective correction based on the orientation of the device.

After determining the amount of perspective correction to perform, theprocess 2400 receives (at 2420) an image captured by a camera of thedual camera mobile device. This process may be performed for each imagein the video sequence captured by the camera. Some embodiments mayperform separate calculations for images coming from each of the twocameras on the dual camera mobile device.

The process then modifies (at 2425) the image based on the determinedamount of perspective correction. Some embodiments also use a baselineimage or other information (e.g., a user-entered point about which thecorrection should be performed) in addition to the warp parameter orother representation of the amount of perspective correction. Aftermodifying the image, process 2400 ends.

FIG. 25 conceptually illustrates example image processing operations ofsome embodiments. This figure illustrates a first image processingoperation 2505 performed by a first image processing module 2520 thatdoes not use perspective correction and a second image processingoperation 2550 performed by a second image processing module 2565 thatuses perspective correction.

As shown, the first image processing operation 2505 is performed on afirst image 2510 of a block 2515 from an aerial perspective lookingdownwards at an angle towards the block. From that perspective, the topof the block 2515 is closer than the bottom of the block. As such, theblock 2515 appears to be leaning towards the camera that captured thefirst image 2510. FIG. 25 also shows the processed first image 2525after processing by the first image processing module 2520. As shown,the block 2515 in the processed first image 2525 appears the samepost-processing, as the first image processing module 2520 did notperform any perspective correction.

The second image processing operation 2550 is performed on a secondimage 2555 of a block 2560. The block 2560 is the same as the block 2515in the first image 2510. FIG. 25 also shows a processed second image2575 after processing of the second image 2555 by the perspectivecorrector 2570 of the second image processing module 2565. Theperspective corrector 2570 may use process 2400 in order to correct theperspective of the second image 2555. Based on data from anaccelerometer and gyroscope indicating that the camera that captured thesecond image 2555 is tilting at a downward angle (and possibly based onother data), the perspective corrector 2570 is able to correct thesecond image so that the block appears to be viewed straight-on in theprocessed second image 2575.

2. Resizing and Bit Stream Manipulation

Among the functions described above by reference to FIG. 21 that areperformed by the image processing layer 930 of some embodiments areimage resizing and bitstream manipulation. Image resizing (performed atoperation 2130) involves scaling up or down an image in some embodiments(i.e., modifying the number of pixels used to represent the image). Insome embodiments, the bitstream manipulation involves inserting datainto the bitstream that indicates the size of the image after resizing.This resizing and bitstream manipulation is performed by an encoderdriver (e.g., driver 1635) in some embodiments.

FIG. 26 conceptually illustrates a software architecture for such anencoder driver 2600 of some embodiments and shows an example resizingand bitstream manipulation operations performed by the encoder driver2600 on an example image 2605. In some embodiments, the image 2605 is animage of a video captured by a camera of the dual camera mobile devicefor transmission to another device(s) in a video conference. Referringto FIG. 16 , in some embodiments the video image will have traveled fromthe CIPU 1650 through the CIPU driver 1630 and camera source module 1622to buffer 1606, from which it is retrieved by image processing manager1608. After undergoing image processing (e.g., focus adjustment,exposure adjustment, perspective correction) in the image processingmanager 1608, the image is sent through buffer 1610 and video compressormodule 1624 to the encoder driver 1635.

As shown, the encoder driver 2600 includes a processing layer 2610 and arate controller 2645. Examples of the rate controller of someembodiments are illustrated in FIG. 30 , described below. The processinglayer 2610 includes an image resizer 2615 and a bitstream manager 2625.In some embodiments, these modules perform various operations on imagesboth before and after the images are encoded. While in this example theimage resizer is shown as part of the processing layer 2610 of theencoder driver 2600, some embodiments implement the image resizer aspart of the image processing manager 1608 rather than the encoder driver2600 (i.e., the image resizing is done before sending the image and thesize data to the encoder driver).

As shown, the image resizer 2615 resizes the images before the imagesare sent to the encoder 2650 through the rate controller 2645. The image2605 is sent through resizer 2615 and scaled down into image 2630. Inaddition to scaling down an image, some embodiments can also scale up animage.

As shown in FIG. 26 , some embodiments scale down the incoming image(e.g., image 2605) and then superimpose the scaled down image (e.g.,image 2630) onto a spatially redundant image (e.g., image 2635) that isthe same size (in pixels) as the incoming image (i.e., the number ofrows and columns of pixels of the image 2605 are the same as the numberof rows and columns of pixels of the spatially redundant image 2635).Some embodiments superimpose the scaled down image 2630 into the upperleft corner of the spatially redundant image (as shown, to producecomposite image 2640), while other embodiments superimpose the scaleddown image into a different section of the spatially redundant image(e.g., the center, upper right, upper center, lower center, lower right,etc.).

In some embodiments, a spatially redundant image is an image that issubstantially all one color (e.g., black, blue, red, white, etc.) or hasa repetitive pattern (e.g., checkers, stripes, etc.). For instance, thespatially redundant image 2635 shown in FIG. 26 has a repetitivecrisscross pattern. The spatially redundant portion of the compositeimage 2640 can be easily compressed by the encoder into a small amountof data due to the repetitive nature. Furthermore, if a sequence ofimages are all scaled down and the spatially redundant image used is thesame for each image in the sequence, then temporal compression can beused to even further reduce the amount of data needed to represent theencoded image.

Some embodiments of the image resizer 2615 also generate size data 2620that indicates the size of the resized image (e.g., the size of thescaled down image 2630) and send this generated size data 2620 to thebitstream manager 2625. The size data 2620 of some embodiments indicatesthe size of the resized image 2630 in terms of the number of rows ofpixels and the number of columns of pixels (i.e., height and width) ofthe resized image 2630. In some embodiments, the size data 2620 alsoindicates the location of the resized image 2630 in the composite image2640.

After the image is resized, the composite image 2640 is sent through therate controller 2645 to the encoder 2650. The rate controller 2645, asdescribed in further detail below, controls the bit rate (i.e., the datasize) of the images output by the encoder 2650 in some embodiments. Theencoder 2650 of some embodiments compresses and encodes the image. Theencoder 2650 may use H.264 encoding or another encoding method.

The bitstream manager 2625 of some embodiments receives a bitstream ofone or more encoded images from the encoder 2650 and inserts size datainto the bitstream. For instance, in some embodiments, the bitstreammanager 2625 receives the size data 2620 from the image resizer 2615 andinserts the size data 2620 into a bitstream 2655 of the encodedcomposite image 2640 that is received from the encoder 2650. The outputof the bitstream manager 2625 in this case is a modified bitstream 2660that includes the size data 2620. Different embodiments insert the sizedata 2620 in different positions of the bitstream 2655. For example, thebitstream 2660 shows the size data 2620 inserted at the beginning of thebitstream 2660. However, other embodiments insert the size data 2620 atthe end of the bitstream 2655, in the middle of the bitstream 2655, orany other position within the bitstream 2655.

In some embodiments, the bitstream 2655 is a bitstream of a sequence ofone or more encoded images that includes the composite image 2640. Insome of such embodiments, the images in the sequence are all resized tothe same size and the size data 2620 indicates the size of those resizedimages. After the images are transmitted to a device on the other end ofthe video conference, the receiving device can extract the sizeinformation from the bitstream and use the size information to properlydecode the received images.

FIG. 27 conceptually illustrates an image resizing process 2700performed by an encoder driver of a dual camera mobile device, such asdriver 2600. The process 2700 begins by receiving (at 2705) an image(e.g., image 2605) captured by a camera of the dual camera mobiledevice. When the dual camera device is capturing images with bothcameras, some embodiments perform process 2700 on images from bothcameras.

Next, the process 2700 resizes (at 2710) the received image. As notedabove, different embodiments resize the image 2605 differently. Forinstance, the image 2605 in FIG. 26 is scaled down and superimposed ontothe spatially redundant image 2635 to produce the composite image 2640.

The process 2700 then sends (at 2715) the resized image (e.g., thecomposite image 2640, which includes the resized image 2630) to theencoder 2650 for encoding. Some embodiments of the process 2700 send theresized image 2630 (included in the composite image 2640) to the encoder2650 through a rate controller that determines a bit rate for theencoder to encode the image. The encoder 2650 of some embodimentscompresses and encodes the image (e.g., using discrete cosine transform,quantization, entropy encoding, etc.) and returns a bitstream with theencoded image to the encoder driver 2600.

Next, the process 2700 sends (at 2720) the data indicating the size ofthe resized image (e.g., the size data 2620) to a bitstream manager. Asshown in FIG. 26 , this operation is performed within the encoder driver2600 in some embodiments (i.e., one module in the encoder driver 2600sends the size data to another module in the encoder driver 2600).

After the resized image is encoded by the encoder 2650, the process 2700receives (at 2725) the bitstream from the encoder. As shown, someembodiments receive the bitstream at the bitstream manager, which alsohas received size data. The received bitstream includes the encodedcomposite image and may also include one or more additional images in avideo sequence.

The process 2700 then inserts (at 2730) the data indicating the size ofthe resized image (e.g., the size data 2620) into the bitstream, andends. As shown in FIG. 26 , this operation is also performed by thebitstream manager in some embodiments. As mentioned above, differentembodiments insert the size data into different parts of the bitstream.In the illustrated example, the size data 2620 is inserted at thebeginning of the bitstream 2655 as shown in the resulting bitstream2660. This bitstream can now be transmitted to another device that isparticipating in the video conference, where it can be decoded andviewed.

In some embodiments, the decoder driver (e.g., driver 1640) performs theopposite functions of the encoder driver. That is, the decoder driverextracts size data from a received bitstream, passes the bitstream to adecoder, and resizes a decoded image using the size data. FIG. 28conceptually illustrates a software architecture for such a decoderdriver 2800 of some embodiments and shows example bitstream manipulationand resizing operations performed by the decoder driver 2800 on anexample bitstream 2825.

In some embodiments, the bitstream 2825 is a bitstream that includes anencoded image of a video captured by a camera of a device in a videoconference (e.g., a bitstream from an encoder driver such as driver2600) and transmitted to the device on which the decoder driver 2800operates. Referring to FIG. 16 , in some embodiments the bitstream willhave been received by the networking manager 1614 and sent to buffer1616, from which it is retrieved by the video decompressor module 1626and sent to the decoder driver 1640.

As shown, the decoder driver 2800 includes a processing layer 2805. Theprocessing layer 2805 includes an image resizer 2810 and a bitstreammanager 2820. In some embodiments, these modules 2810 and 2820 performvarious operations on received images both before and after the imagesare decoded. While in this example the image resizer 2810 is shown aspart of the processing layer 2805 of the decoder driver 2800, someembodiments implement the image resizer as part of the image processingmanager 1608 rather than the decoder driver (i.e., the image resizing isdone after sending the image from the decoder driver 2800).

As shown, the bitstream manager 2820 of some embodiments receives abitstream of one or more encoded images (i.e., images in a videosequence) and extracts size data from the bitstream before sending thebitstream to the decoder 2835 for decoding. For example, as illustratedin FIG. 28 , the bitstream manager 2820 receives a bitstream 2825 of anencoded image, extracts a size data 2815 from the bitstream 2825, andsends the resulting bitstream 2830 (without the size data 2815) to thedecoder 2835 for decoding. As shown, the bitstream manager 2820 sendsthe extracted size data 2815 to the image resizer 2810 in someembodiments.

The size data 2815 of some embodiments is the same as the size data 2620inserted into the bitstream by the encoder driver 2600. As describedabove in the description of FIG. 26 , the size data 2815 of someembodiments indicates the size of a sub-image 2845 in terms of thenumber of rows of pixels and the number of columns of pixels of thesub-image 2845. The size data 2815 may also indicate the location of thesub-image 2845 within the larger spatially redundant image 2840. In thisexample, the bitstream 2825 shows the size data 2815 inserted at thebeginning of the bitstream 2825. However, as noted above, differentembodiments insert the size data 2815 in different positions of thebitstream 2825.

The image resizer 2810 of some embodiments extracts sub-images fromimages using size data received from the bitstream manager 2820. Forinstance, FIG. 28 illustrates the image resizer 2810 receiving an image2840 that includes a sub-image 2845 from the decoder 2835. As shown, theimage resizer 2810 of some embodiments extracts the sub-image 2845 fromthe image 2840. This extracted image can then be displayed on the dualcamera mobile device.

FIG. 29 conceptually illustrates an image extraction process 2900 ofsome embodiments performed by a decoder driver of a device participatingin a video conference, such as driver 2800. The process begins byreceiving (at 2905) a bitstream (e.g., bitstream 2825) of an encodedimage. The bitstream may be sent from a device participating in a videoconference with the device on which the decoder driver is operating ormay be stored in a storage of the device. When the device is receivingimages from multiple sources, some embodiments perform process 2900 onimages from each source.

Next, the process 2900 extracts (at 2910) size data from the bitstream.As noted above, this size data may be found in different locations inthe bitstream. Some embodiments know where to look for the size data,while other embodiments look for a particular signature that indicateswhere in the received bitstream the size data is located. In someembodiments, the size data indicates the size (e.g., the number ofpixels in each row and number of pixels in each column) and the locationof a sub-image in the encoded image.

The process 2900 then sends (at 2915) the extracted size data to animage resizer. As shown in FIG. 28 , this operation is performed withinthe decoder driver in some embodiments (i.e., one module in the decoderdriver sends the size data to another module in the decoder driver).

The process 2900 also sends (at 2920) the bitstream to the decoder fordecoding. The decoder, in some embodiments decompresses and decodes thebitstream (e.g., using inverse discrete cosine transform, inversequantization, etc.) and returns a reconstructed image to the decoderdriver.

After the bitstream is decoded by the decoder, the process 2900 receives(at 2925) the decoded image from the decoder. As shown, some embodimentsreceive the image at the image resizer, which also has received sizedata from the bitstream manager. The process then extracts (at 2930) asub-image from the decoded image using the received size data. As shown,the sub-image 2845 is extracted from the upper left of decoded image2840, as indicated in size data 2815. This extracted sub-image can nowbe displayed on a display device (e.g., a screen of the dual cameramobile device).

3. Rate Controllers

In some embodiments, the two cameras of the device have different setsof characteristics. For example, in some embodiments, the front camerais a lower resolution camera optimized for the capture of motion videoimages while the back camera is a higher resolution camera optimized forthe capture of still images. For reasons such as cost, functionality,and/or geometry of the device, other embodiments may use differentcombinations of cameras of different characteristics.

Cameras with different characteristics can introduce differentartifacts. For example, higher resolution cameras may reveal more noisethan lower resolution cameras. Images captured by higher resolutioncameras may exhibit higher levels of spatial or temporal complexitiesthan images captured by lower resolution cameras. Also, differentcameras with different optical properties may introduce different gammavalues to the captured images. Different light sensing mechanisms usedby different cameras to capture images may also introduce differentartifacts.

Some of these camera-specific artifacts conceal artifacts generated fromother sources. For example, in an image captured by a high resolutioncamera with a high level of noise, artifacts that are the byproduct ofthe video encoding process become less visible. When encoding noise(such as quantization distortion) to hide behind camera-specificartifacts, the video encoding process can use larger quantization stepsizes to achieve lower bit rates. On the other hand, when a cameraintroduces less artifacts (such as in the case of a lower resolutioncamera), the video encoding process can use finer quantization stepsizes in order to avoid unacceptable levels of visual distortion due toquantization. Thus, a video encoding process that is optimized to takeadvantage of or to compensate for these camera-specific characteristicscan accomplish better rate-distortion trade-off than the video encodingprocess that is oblivious to these camera-specific characteristics.

In order to utilize these camera-specific characteristics for performingrate-distortion trade-offs, some embodiments implement two videoencoding processes, each process optimized to each of the two cameras.FIG. 30 illustrates an example of a system with two video encodingprocesses for two cameras 3060 and 3070. As shown in FIG. 30 , thesystem 3000 includes encoder driver 3010, rate controllers 3020 and3040, and a video encoder 3030. The encoder 3030 encodes video imagescaptured from video cameras 3060 and 3070 into bitstreams 3080 and 3090.

In some embodiments, the video encoder driver 3010 is a software modulerunning on one or more processing units. It provides an interfacebetween the video encoder 3030 and other components of the system, suchas video cameras, image processing modules, network management modulesand storage buffers. The encoder driver 3010 controls the flow ofcaptured video image from the cameras and the image processing modulesto the video encoder 3030, and it also provides the conduit for theencoded bitstreams 3080 and 3090 to storage buffers and networkmanagement modules.

As shown in FIG. 30 , the encoder driver 3010 includes two differentinstances 3020 and 3040 of rate controllers. These multiple instancescan be two different rate controllers for the two different cameras, orone rate controller that is configured in two different manners for twodifferent cameras. Specifically, in some embodiments, the two ratecontrollers 3020 and 3040 represent two separate rate controllers.Alternatively, in other embodiments, the two rate controllers 3020 and3040 are two different configurations of a single rate controller.

FIG. 30 also shows the encoder driver 3010 to include a state buffer3015 that stores encoding state information for the rate controllingoperations to use during a video conference. Specifically, in someembodiments, the two different rate controllers, or the two differentconfigurations of the same rate controller, share during a videoconference the same encoding state information that is stored in thestate buffer 3015. Such sharing of state information allows uniform ratecontroller operations in dual video capture video conferences. Thissharing also allows optimal video encoding during a switch cameraoperation in a single video capture video conference (i.e., allows therate controlling operation for the encoding of video captured by thecurrent camera to use encoding state information that was maintained bythe rate controlling operation for the encoding of the video captured bythe previous camera). FIG. 30 shows the state buffer 3015 as being partof the encoder driver 3010, but other embodiments implement the statebuffer 3015 outside the encoder driver 3010.

In the state buffer 3015, different embodiments store different types ofdata (e.g., different types of encoding parameters) to represent theencoding state information. One example of such encoding stateinformation is the current target bit rate for the video conference. Onemanner for identifying the target bit rate is described above in SectionIII.B. Other examples of such encoding state information include bufferfullness, maximum buffer fullness, bit rates of one or more recentlyencoded frames, among other encoding state information.

A rate controller can then use the target bit rate (or another encodingstate parameter stored in the state buffer) to calculate one or moreparameters used in its rate controlling operation. For instance, asfurther described below, a rate controller of some embodiments uses thecurrent target bit to calculate a quantization parameter QP for amacroblock or a frame. By way of example, some embodiments use thecurrent target bit rate to compute a quantization adjustment parameterfrom which they derive the quantization parameter QP for the macroblockand/or the frame. Accordingly, during a camera switch operation in avideo conference, sharing the target bit rate between the two ratecontrolling operations (of two rate controllers or of two differentconfigurations of one rate controller) allows the rate controllingoperation for encoding the video captured by the current camera to getthe benefit of the encoding state data from the previous ratecontrolling operation for encoding the video captured by the previouscamera.

FIG. 30 illustrates the encoder driver 3010 to include the two differentrate-controller instances 3020 and 3040. However, in other embodiments,these rate controller instances 3020 and 3040 are built into videoencoder 3030. The video encoder 3030 encodes video images captured bythe cameras 3060 and 3070 into digital bitstreams 3080 and 3090. In someembodiments, the video encoder produces bitstreams that are compliantwith conventional video coding standards (e.g., H.264 MPEG-4). In someof these embodiments, the video encoder performs encoding operationsthat include motion estimation, discrete cosine transform (“DCT”),quantization, and entropy encoding. The video encoder also performsdecoding operations that are the inverse functions of the encodingoperations.

In some embodiments, the encoder 3030 includes a quantizer module 3032for performing quantization. The quantizer module is controlled by aquantization parameter 3022 or 3042 from a rate controller 3020 or 3040.In some embodiments, each quantization parameter is set by acorresponding rate controller and is a function of one or moreattributes of the camera associated with the rate controller, as furtherdescribed below. The rate controller can reduce the number of bits usedfor encoding by setting coarser quantization step sizes or increase thenumber of bits used by setting finer quantization step sizes. Bycontrolling the quantization step size, the rate controller alsodetermines how much distortion is introduced into the encoded videoimage. Thus the rate controller can perform trade-offs between bit rateand image quality. In performing the rate-distortion trade off, the ratecontroller monitors bit rate in order not to overflow memory buffers,underflow memory buffers, or exceed the transmission channel capacity.The rate controller must also control bit rate in order to provide thebest possible image quality and to avoid unacceptable distortion ofimage quality due to quantization. In some embodiments, each ratecontroller stores the monitored data in terms of a set of state datavalues in the state buffer 3015. In some embodiments, the ratecontrollers 3020 and 3040 uses camera-specific attributes to optimizerate-distortion trade off.

In some embodiments, each rate controller optimizes rate-distortiontrade off by directly applying a modification factor to its quantizationparameter. In some of these embodiments, the modification factors arepre-determined and built into the device along with the camera; thedevice does not need to dynamically compute these modification factors.In other embodiments, the system uses the incoming image captured by thecamera to dynamically determine the appropriate modification factorspecific to the camera. In some of these embodiments, the systemanalyzes a sequence of incoming video images captured by the camera inmultiple encoding passes in order to collect certain statistics aboutthe camera. The system then uses these statistics to derive modificationfactors to the quantization parameter that is optimized for the camera.

In some embodiments, these camera-specific modification factors areapplied to the quantization parameter via visual masking attributes ofthe video images. Visual masking attribute of an image or a portion ofthe image is an indication of how much coding artifacts can be toleratedin the image or image portion. Some embodiments compute a visual maskingattribute that quantifies the brightness energy of the image or theimage portion while other embodiments compute a visual masking attributethat quantifies the activity energy or complexity of the image or theimage portion. Regardless of how a visual masking attribute iscalculated, some embodiments use visual masking attributes to calculatea modified or masked quantization parameter for a video frame. Some ofthese embodiments calculate the masked quantization parameter as afunction of a frame level visual masking attribute φ_(frame) and areference visual masking attribute φ_(R). In some embodiments, thequantization parameter modified by visual masking attributes φ_(frame)and φ_(R) is expressed as:

MQP _(frame) =QP _(nom)+β_(frame)*(φ_(frame)−φ_(R))/φ_(R)  (1)

where MQP_(frame) is masked or modified quantization parameter for theframe, QP_(nom) is an initial or nominal quantization value, andβ_(frame) is a constant adapted to local statistics. In someembodiments, the reference visual masking attribute φ_(R) and nominalquantization parameter QP_(nom) are pre-determined from an initial orperiodic assessment of network conditions.

In some embodiments, the visual masking attribute φ_(frame) in equation(1) is calculated as

φ_(frame) =C·(E·avgFrameLuma)^(β)·(D·avgFrameSAD)^(α)  (2)

where avgFrameLuma is the average luminance value of the frame andavgFrameSAD is the average sum of absolute difference of the frame.Constants a, p, C, D, and E are adapted to local statistics. Theseconstants are adapted to camera specific characteristics in someembodiments.

Some embodiments also calculate a masked quantization parameter for aportion of a video image such as a macroblock. In those instances, themasked quantization parameter is calculated as a function of themacroblock visual masking attribute φ_(MB):

MQP _(MB) =MQP _(frame)+β_(MB)*(φ_(MB)−φ_(frame))/φ_(frame)  (3)

where β_(MB) is a constant adapted to local statistics, and MQP_(frame)is calculated using equations (1) and (2) in some embodiments. In someembodiments, the visual masking attribute φ_(MB) in equation (3) iscalculated as

φ_(MB) =A·(C·avgMBLuma)^(β)·(B·MBSAD)^(α)  (4)

where avgMBLuma is the average luminance value of the macroblock andavgMBSAD is the average sum of absolute difference of the macroblock.Constants α, β, A, B and C are adapted to local statistics. Theseconstants are adapted to camera specific characteristics in someembodiments.

Rather than using multiple camera-specific constants to compute themodified quantization parameters as discussed above, some embodimentsperform camera-specific rate control by computing quantizationparameters using only a single camera-specific coefficient. For example,given visual masking attributes φ_(frame) and φ_(MB) and quantizationparameter QP_(frame), some embodiments use a single camera-specificcoefficient μ to calculate the quantization parameter of a macroblockas:

QP _(MB)=μ·(φ_(frame)−φ_(MB))+QP _(frame)  (5)

To compute equation (5), some embodiments use complexity measures of theframe and of the macroblock as visual masking attributes φ_(frame) andφ_(MB), respectively.

Some embodiments apply a different camera specific coefficient in thecalculation of QP_(MB). For example, in some embodiments, QP_(MB) iscalculated as

QP _(MB)=ρ·(1−φ_(MB)/φ_(frame))·QP _(frame) +QP _(frame)  (6)

where ρ is a coefficient tuned to camera-specific characteristics.

As mentioned above, the state buffer 3015 stores encoding stateinformation that the two different rate controller instances 3020 and3040 can share during a video conference in order to obtain betterencoding results from their rate controlling operations. Target bit rateR_(T) is one example of such shared state information in someembodiments. This rate is a desired bit rate for encoding a sequence offrames. Typically, this bit rate is expressed in units of bits/second,and is determined based on processes like those described above inSection III.B.

As described above, a rate controller of some embodiments uses thetarget bit rate to calculate the frame and/or macroblock quantizationparameter(s) QP that it outputs to the video encoder 3030. For example,some embodiments use the current target bit rate to compute aquantization adjustment parameter from which they derive thequantization parameter QP for the macroblock and/or the frame. In someembodiments, the quantization adjustment parameter is expressed in termsof a fraction that is computed by dividing either the previous frame'sbit rate or a running average of the previous frames' bit rate, with thecurrent target bit rate. In other embodiments, this adjustment parameteris not exactly computed in this manner, but rather is more generally (1)proportional to either the previous frame's bit rate or a runningaverage of the previous frames' bit rate, and (2) inversely proportionalto the current target bit rate.

After computing such a quantization adjustment parameter, the ratecontroller of some embodiments uses this parameter to adjust themacroblock and/or frame quantization parameter(s) that it computes. Onemanner of making such an adjustment is to multiply the computedmacroblock and/or frame quantization parameter(s) by the quantizationadjustment parameter. Another manner of making this adjustment is tocompute an offset quantization parameter value from the quantizationadjustment parameter and then apply (e.g., subtract) this offsetparameter to the computed macroblock and/or frame quantizationparameter(s). The rate controller of these embodiments then outputs theadjusted macroblock and/or frame quantization parameter(s) to the videoencoder 3030.

In other embodiments, the rate controller uses the target bit rate tocalculate other parameters that are used in its rate controllingoperation. For instance, in some embodiments, the rate controller usesthis bit rate to modify the visual masking strength for a macroblock ora frame.

G. Networking Manager

FIG. 31 conceptually illustrates the software architecture of anetworking manager 3100 of some embodiments such as the networkingmanager 1614 illustrated in FIG. 16 . As described above, the networkingmanager 3100 manages network connections (e.g., connectionestablishment, connection monitoring, connection adjustments, connectiontear down, etc.) between a dual camera mobile device on which itoperates and a remote device in a video conference. During the videoconference, the networking manager 3100 of some embodiments alsoprocesses data for transmission to the remote device and processes datareceived from the remote device.

As shown in FIG. 31 , the networking manager 3100 includes a sessionnegotiating manager 3105, a transmitter module 3115, a universaltransmission buffer 3120, a universal transmission buffer manager 3122,a virtual transport protocol (VTP) manager 3125, a receiver module 3130,and a media transport manager 3135.

The session negotiating manager 3105 includes a protocol manager 3110.The protocol manager 3110 ensures that the transmitter module 3115 usesa correct communication protocol to transmit data to a remote deviceduring the video conference and enforces rules of the communicationprotocol that is used. Some embodiments of the protocol manager 3110support a number of communication protocols, such as a real-timetransport protocol (RTP), a transmission control protocol (TCP), a userdatagram protocol (UDP), and a hypertext transfer protocol (HTTP), amongothers.

The session negotiating manager 3105 is responsible for establishingconnections between the dual camera mobile device and one or more remotedevices participating in the video conference, as well as tearing downthese connections after the conference. In some embodiments, the sessionnegotiating manager 3105 is also responsible for establishing multimediacommunication sessions (e.g., to transmit and receive video and/or audiostreams) between the dual camera mobile device and the remote devices inthe video conference (e.g., using a session initiation protocol (SIP)).

The session negotiating manager 3105 also receives feedback data fromthe media transport manager 3135 and, based on the feedback data,determines the operation of the universal transmission buffer 3120(e.g., whether to transmit or drop packets/frames) through the universaltransmission buffer manager 3122. This feedback, in some embodiments,may include one-way latency and a bandwidth estimation bit rate. Inother embodiments, the feedback includes packet loss information androundtrip delay time (e.g., determined based on packets sent to theremote device in the video conference and the receipt ofacknowledgements from that device). Based on the information from themedia transport manager 3135, the session negotiating manager 3105 candetermine whether too many packets are being sent and instruct theuniversal transmission buffer manager 3122 to have the universaltransmission buffer 3120 transmit fewer packets (i.e., to adjust the bitrate as described in FIG. 15 ).

The transmitter module 3115 retrieves encoded images (e.g., as abitstream) from a video buffer (e.g., the buffer 1612 of FIG. 16 ) andpacketizes the images for transmission to a remote device in the videoconference through the universal transmission buffer 3120 and thevirtual transport protocol manager 3125. The manner in which the encodedimages are created and sent to the transmitter module 3115 can be basedon instructions or data received from the media transport manager 3135and/or the session negotiating manager 3105. In some embodiments,packetizing the images involves breaking the received bitstream into agroup of packets each having a particular size (i.e., a size specifiedby the session negotiating manager 3105 according to a particularprotocol), and adding any required headers (e.g., address headers,protocol specification headers, etc.).

The universal transmission buffer manager 3122 controls the operation ofthe universal transmission buffer 3120 based on data and/or instructionsreceived from the session negotiating manager 3105. For example, theuniversal transmission buffer manager 3122 may be instructed to directthe universal transmission buffer 3120 to transmit data, stoptransmitting data, drop data, etc. As described above, in someembodiments when a remote device participating in the conference appearsto be dropping packets, this will be recognized based onacknowledgements received from the remote device. To reduce the packetdropping, the universal transmission buffer manager 3122 may beinstructed to transmit packets at a slower rate to the remote device.

The universal transmission buffer 3120 stores data received from thetransmitter module 3115 and transmits the data to the remote devicethrough the VTP manager 3125. As noted above, the universal transmissionbuffer 3120 may drop data (e.g., images of the video) based oninstructions received from the universal transmission buffer manager3122.

In some embodiments, RTP is used to communicate data packets (e.g.,audio packets and video packets) over UDP during a video conference.Other embodiments use RTP to communicate data packets over TCP duringthe video conference. Other transport layer protocols can be used aswell in different embodiments.

Some embodiments define a particular communication channel between twomobile devices by a pair of port numbers (i.e., source port number anddestination port number). For instance, one communication channelbetween the mobile devices can be defined by one pair of port numbers(e.g., source port 50 and destination port 100) and another differentcommunication channel between the mobile devices can be defined byanother different pair of port numbers (e.g., source port 75 anddestination port 150). Some embodiments also use a pair of Internetprotocol (IP) addresses in defining communication channels. Differentcommunication channels are used to transmit different types of datapackets in some embodiments. For example, video data packets, audio datapackets, and control signaling data packets can be transmitted inseparate communication channels. As such, a video communication channeltransports video data packets and an audio communication channeltransports audio data packets.

In some embodiments, a control communication channel is for messagingbetween the local mobile device and a remote device during a videoconference. Examples of such messaging include sending and receivingrequests, notifications, and acknowledgements to such requests andnotifications. Another example of messaging includes sending remotecontrol instruction messages from one device to another. For instance,the remote control operations described below (e.g., instructing adevice to only send images from one particular camera or to only captureimages with a particular camera) can be performed by sendinginstructions from a local device to a remote device through the controlcommunication channel for the local device to remotely controloperations of the remote device. Different embodiments implement thecontrol communication using different protocols like a real-timetransport control protocol (RTCP), an RTP extension, SIP, etc. Forinstance, some embodiments use RTP extension to relay one set of controlmessages between two mobile devices in a video conference and use SIPpackets to relay another set of control messages between the mobiledevices during the video conference.

The VTP manager 3125 of some embodiments allows different types of datapackets that are specified to be transmitted through differentcommunication channels (e.g., using different pairs of port numbers) tobe transmitted through a single communication channel (e.g., using thesame pair of port numbers). One technique for doing this involvesidentifying the data packet types, identifying the communication channelthrough which data packets are specified to be transmitted by extractingthe specified pair of port numbers of the data packets, and specifyingthe data packets to be transmitted through the single communicationchannel by modifying the pair of port numbers of the data packets to bethe pair of port numbers of the single communication channel (i.e., allthe data packets are transmitted through the same pair of port numbers).

To keep track of the original pair of port numbers for each type of datapacket, some embodiments store a mapping of the original pair of portnumbers for the data packet type. Some of these embodiments than use thepacket type field of the protocol to differentiate the different packetsthat are being multiplexed into one communication channel. For instance,some embodiments that have the VTP manager multiplex audio, video andcontrol packets into one RTP stream, use the RTP packet type field todifferentiate between the audio, video and control packets that aretransmitted in the one RTP channel to the other device in the videoconference. In some of these embodiments, the VTP manger also routescontrol messaging in SIP packets to the other device.

Some embodiments identify examine the data packet signatures (i.e.,packet header formats) to distinguish between different packets that arecommunicated using different protocols (e.g., to differentiate betweenpackets transported using RTP and packets transported using SIP). Insuch embodiments, after the data packets of the different protocols aredetermined, the fields of the data packets that use the same protocol(e.g., audio data and video data using RTP) are examined as describedabove to identify the different data types. In this manner, the VTPmanager 3125 transmits different data packets, which are intended to betransmitted through different communication channels, through a singlecommunication channel.

Although one way of combining different types of data through a singlecommunication channel is described above, other embodiments utilizeother techniques to multiplex different packet types into onecommunication stream. For example, one technique of some embodimentsinvolves keeping track of the original pair of port numbers of the datapackets and storing the original pair of port numbers in the data packetitself to be later extracted. Still other ways exist for combiningdifferent types of data between two video conference participants intoone port pair channel.

When the VTP manager 3125 receives data packets from the remote devicethrough a virtualized communication channel, the VTP manager 3125examines the signatures of the data packets to identify the differentpackets that are sent using the different protocols. Such signatures canbe used to differentiate SIP packets from RTP packets. The VTP managerof some embodiments also uses the packet type field of some or all ofthe packets to demultiplex the various different types of packets (e.g.,audio, video and control packets) that were multiplexed into a singlevirtualized channel. After identifying these different types of packets,the VTP manager associates each different type of packet with itscorresponding port pair numbers based on a mapping of port pair numbersand packet types that it keeps. The VTP manager 3125 then modifies thepair of port numbers of the data packets with the identified pair ofport numbers and forwards the data packets to be depacketized. In otherembodiments that use different techniques for multiplexing differentpacket types into the single channel, the VTP manager uses differenttechniques for parsing out the packets.

By using such techniques for multiplexing and de-multiplexing thedifferent packets, the VTP manager 3125 creates a single virtualizedcommunication channel (e.g., a single pair of port numbers), transmitsthe video data, audio data, and control signaling data through thesingle virtualized communication channel, and receives audio, video, andcontrol packets from the remote device through the single virtualizedcommunication channel. Thus, from the perspective of the network, datais transmitted through this single virtualized communication channel,while, from the perspective of the session negotiating manager 3105 andthe protocol manager 3110, the video data, audio data, and controlsignaling data are transmitted through different communication channels.

Similar to the images that are transmitted to the remote device in thevideo conference, images transmitted from the remote device in the videoconference are received in packet format. The receiver module 3130receives the packets and depacketizes them in order to reconstruct theimages before storing the images in a video buffer (e.g., the buffer1616 of FIG. 16 ) to be decoded. In some embodiments, depacketizing theimages involves removing any headers and reconstructing a bitstream thatonly has image data (and potentially size data) from the packets.

The media transport manager 3135 processes feedback data (e.g., one-waylatency, bandwidth estimation bit rate, packet loss data, roundtripdelay time data, etc.) received from the network to dynamically andadaptively adjust the rate of data transmission (i.e., bit rate). Themedia transport manager 3135 also controls error resilience based on theprocessed feedback data in some other embodiments, and may also send thefeedback data to the video conference manager 1604 in order to adjustother operations of the video conference module 1602 such as scaling,resizing, and encoding. In addition to having the universal transmissionbuffer drop packets when a remote device in the conference is not ableto process all of the packets, the video conference module and encodercan use a lower bit rate for encoding the images so that fewer packetswill be sent for each image.

In some embodiments, the media transport manager 3135 may also monitorother variables of the device such as power consumption and thermallevels that may affect how the operational power modes of the camerasare configured, as discussed above. This data may also be used asadditional inputs into the feedback data (e.g., if the device is gettingtoo hot, the media transport manager 3135 may try to have the processingslowed down).

Several example operations of the networking manager 3100 will now bedescribed by reference to FIG. 16 . The transmission of images capturedby a camera of the dual camera mobile device to a remote device in thevideo conference will be described first, followed by the description ofreceiving images from the remote device. The transmitter module 3115retrieves encoded images from the buffer 1612, which are to betransmitted to the remote device in the video conference.

The protocol manager 3110 determines the appropriate protocol to use(e.g., RTP to transmit audio and video) and the session negotiatingmanager 3105 informs the transmitter module 3115 of such protocol. Next,the transmitter module 3115 packetizes the images and sends thepacketized images to the universal transmission buffer 3120. Theuniversal transmission buffer manager 3122 receives instructions fromthe session negotiating manager 3105 to direct the universaltransmission buffer 3120 to transmit or drop the images. The VTP manager3125 receives the packets from the universal transmission buffer 3120and processes the packets in order to transmit the packets through asingle communication channel to the remote device.

When receiving images from the remote device, the VTP manager 3125receives packetized images from the remote device through thevirtualized single communication channel and processes the packets inorder to direct the images to the receiver module 3130 through acommunication channel that is assigned to receive the images (e.g., avideo communication channel).

The receiver module 3130 depacketizes the packets to reconstruct theimages and sends the images to the buffer 1616 for decoding by thedecoder 1660. The receiver module 3130 also forwards control signalingmessages to the media transport manager 3135 (e.g., acknowledgements ofreceived packets from the remote device in the video conference).

Several example operations of the networking manager 3100 were describedabove. These are only illustrative examples, as various otherembodiments will perform these or different operations using differentmodules or with functionalities spread differently between the modules.Furthermore, additional operations such as dynamic bit rate adjustmentmay be performed by the modules of networking manager 3100 or othermodules.

IV. In-Conference Adjustment and Control Operations

A. Picture-In-Picture Modifications

1. Snap-to-Corner

Some embodiments of the invention allow a user of a dual camera mobiledevice to modify a composite display displayed on the device by movingaround one or more display areas that form the composite display. Onesuch example is moving around an inset display area of a PIP display.FIG. 32 illustrates such an example that is performed during a videoconference. In a video conference, the user may want to move aforeground inset display area for a variety of reasons, such as whenthis area is blocking an area of interest of the background displayarea.

FIG. 32 illustrates the moving of an inset display area 3240 in a UI3205 of a device, by reference to five different stages 3210, 3215,3220, 3225, and 3230 of this UI. The first stage 3210 illustrates the UI3205 during a video conference between the local user of the device anda remote user of a remote device. The UI 3205 in FIG. 32 shows a PIPdisplay that is the same PIP display shown in the fifth stage of FIG. 11after the video conference has started. In this example, the videocaptured by the local user's device is displayed in the inset displayarea 3240 and the video captured by the remote user's device isdisplayed in the background display area 3235. As shown, the displayarea 1155 includes a selectable UI item 3245 for ending the videoconference. In some embodiments, the layout of the display area 1155 isthe same as the layout of the display area 1155 of FIG. 12 , describedabove.

The second stage 3215 illustrates the user starting a snap-to-corneroperation by selecting the inset display area 3240. In this example, aselection is made by placing a finger 3255 anywhere within the insetdisplay area 3240. As shown, this selection is displayed in terms of athick border 3260 for the inset display area 3240. Different embodimentsmay indicate such a selection in different ways, such as by highlightingthe display area 3240, by causing the display area 3240 to vibrate, etc.

The third stage 3220 illustrates the UI 3205 after the user begins tomove the inset display area 3240 of the PIP display 3250 from one areain the PIP display 3250 to another area in this display. In thisexample, the inset display area 3240 has started to move from the lowerleft corner of the PIP display 3250 to the lower right corner of thisdisplay, as indicated by the arrow 3265. In this example, the insetdisplay area 3240 is moved by the user dragging his finger 3255 towardsthe lower right corner of the PIP display 3250 after selecting the insetdisplay in the second stage 3215. Some embodiments provide othertechniques for moving the inset display area 3240 around in the PIPdisplay 3250.

The fourth stage 3225 illustrates the UI 3205 in a state after the userhas removed his finger 3255 from the screen of the device 3200. In thisstate, the inset display area 3240 is still moving towards the lowerright corner of the PIP display 3250 that was identified based on theuser's finger movement in the third stage 3220. In other words, afterthe finger 3255 starts the movement of the inset display area 3240towards the lower right corner of the PIP display 3250, the UI 3205maintains this movement even after the finger 3255 is removed. Tomaintain this movement, the UI 3205 of some embodiments requires theuser's drag operation to be larger than a particular threshold amount(e.g., longer than a particular distance or longer than a particularlength of time) before the user removes his finger 3255; otherwise,these embodiments keep the inset display area 3240 in its original leftcorner position after moving this display area 3240 slightly or notmoving it at all.

However, while some embodiments allow the inset display area to moveeven after the user stops his drag operation before the inset displayarea has reached its new location, other embodiments require the user tomaintain his drag operation until the inset display area reaches its newlocation. Some embodiments provide still other techniques for moving theinset display area. For example, some embodiments may require the userto specify where to direct the inset display area 3240 before the insetdisplay area 3240 actually starts to move, etc. Some embodiments mayalso allow display areas to slide and snap-to-corners by simply tiltingthe mobile device at different angles.

The fifth stage 3230 illustrates the UI 3205 after the inset displayarea 3240 has reached its new location at the bottom right corner of thePIP display 3250. The removal of the thick border 3260 in the fifthstage 3230 indicates that the snap-to-corner operation is completed.

To facilitate the movement illustrated in the above-described third,fourth and fifth stages 3220, 3225 and 3230, the UI 3205 of someembodiments employ snapping rules that allow the inset display area toquickly snap to a corner of the PIP display 3250 once the user causesthe inset display area to move towards that corner. For instance, whenthe user drags the inset display area 3240 by more than a thresholdamount towards a particular corner, the UI 3205 of some embodimentsidentifies the direction of motion of the inset display area 3240,determines that the motion has exceeded a threshold amount, and thensubsequently moves the inset display area 3240 automatically withoutfurther user input to the next grid point in the UI 3205 to which theinset display area 3240 can be snapped. In some embodiments, the onlygrid points that are provided for snapping the inset display area 3240are grid points at the four corners of the PIP display 3250. Otherembodiments provide other grid points in the UI 3205 (e.g., in the PIPdisplay 3250) to which the inset display area 3240 can snap (i.e., towhich the sides or vertices of the area 3240 can be placed on or alignedwith).

Still other embodiments may not employ grid points so that the insetdisplay area can be positioned at any point in the PIP display 3250. Yetother embodiments provide a feature that allows the user to turn on oroff the snap to grid point feature of the UI. Moreover, in addition tothe video captured from the devices, different embodiments may allow theuser to perform the snap-to-corner operation with various items, such asicons, etc.

FIG. 33 illustrates two other examples 3330 and 3335 of a snap-to-corneroperation in the UI 3205. These other snap-to-corner operations show theinset display area 3240 being moved vertically or diagonally in the PIPdisplay 3250, based on vertical or diagonal dragging operations of theuser.

Even though FIGS. 32 and 33 illustrate the movement of the inset displayarea within a PIP display, one of ordinary skill will realize that otherembodiments utilize similar techniques to move display areas in othertypes of PIP displays or other types of composite displays. Forinstance, as further described below, the PIP display of someembodiments has two or more foreground inset displays and these insetdisplays can be moved in the PIP display using techniques similar tothose described above by reference to FIGS. 32 and 33 . Also, someembodiments use similar techniques to move around display areas incomposite displays (e.g., to move one display area from a left side ofthe screen to the right side of the screen through a user dragmovement). Furthermore, the moving of a display area(s) of a compositedisplay can cause changes to the image processing operations of the dualcamera mobile device such as causing the video conference manager 1604to re-composite the display area in the composite display in response tothe user's input. As further described below, some embodiments employsnap and push techniques that push a first display area from a firstlocation when a second display area is moved to the first location froma third location.

2. Rotate

Some embodiments rotate the PIP display that is presented during a videoconference when a user of the mobile device used for the videoconference rotates the device during the conference. FIG. 34 illustratesthe rotation of a UI 1105 of a device 3400 when the device is rotatedfrom a vertical position to a horizontal position. The device 3400 isheld vertically when the long side of the screen is vertical whereas thedevice 3400 is held horizontally when the long side of the screen ishorizontal. In the example illustrated in FIG. 34 , the UI 1105 rotatesfrom a portrait view that is optimized for a vertical holding of thedevice to a landscape view that is optimized for horizontal holding ofthe device 3400. This rotation functionality allows the user to view theUI 1105 displayed in an upright position when the mobile device 3400 isheld either vertically or horizontally.

FIG. 34 illustrates the rotation of the UI 1105 in terms of sixdifferent operational stages 3410, 3415, 3420, 3425, 3430 and 3435. Thefirst stage 3410 illustrates the UI 1105 during a video conferencebetween the local user of the device and a remote user of a remotedevice. The UI 1105 in FIG. 34 shows a PIP display 1180 that is the samePIP display shown in the fifth stage of FIG. 11 after the videoconference has been established. In this example, the video captured bythe local user's device is displayed in the inset display area 1160 andthe video captured by the remote user's device is displayed in thebackground display area 1170. In the display area 1155 below the PIPdisplay 1180 includes a selectable UI item 3485 (e.g., an End Conferencebutton 3485), which the user may select to end the video conference(e.g., through a single finger tap).

The second stage 3415 illustrates the UI 1105 after the user begins totilt the device 3400 sideways. In this example, the user has started totilt the device 3400 from being held vertically to being heldhorizontally, as indicated by the arrow 3460. The appearance of the UI1105 has not changed. In other situations, the user may want to tilt thedevice 3400 from being held horizontally to being held verticallyinstead, and, in these situations, the UI 1105 switches from ahorizontally optimized view to a vertically optimized view.

The third stage 3420 illustrates the UI 1105 in a state after the device3400 has been tilted from being held vertically to being heldhorizontally. In this state, the appearance of the UI 1105 still has notchanged. In some embodiments, the rotation operation is triggered afterthe device 3400 is tilted past a threshold amount and is kept past thispoint for a duration of time. In the example illustrated in FIG. 34 , itis assumed that the threshold amount and the speed of the rotation donot cause the UI 1105 to rotate until a short time interval after thedevice has been placed in the horizontal position. Different embodimentshave different threshold amounts and waiting periods for triggering therotation operation. For example, some embodiments may have such a lowthreshold to triggering the rotation operation as to make the UI 1105appear as if it were always displayed in an upright position,notwithstanding the orientation of the device 3400. In otherembodiments, the user of the device 3400 may specify when the rotationoperation may be triggered (e.g., through a menu preference setting).Also, some embodiments may not delay the rotation after the device istilted past the threshold amount. Moreover, different embodiments mayallow the rotation operation to be triggered in different ways, such asby toggling a switch on the mobile device, by giving voice commands,upon selection through a menu, etc.

The fourth stage 3425 illustrates the UI 1105 after the rotationoperation has started. Some embodiments animate the rotation displayareas to provide feedback to the user regarding the rotation operation.FIG. 34 illustrates an example of one such animation. Specifically, itshows in its fourth stage 3425 the start of the rotation of the displayareas 1180 and 1155 together. The display areas 1180 and 1155 rotatearound an axis 3465 going through the center of the UI 1105 (i.e., thez-axis). The display areas 1180 and 1155 are rotated the same amount butin the opposite direction of the rotation of the device 3400 (e.g.,through the tilting of the device 3400). In this example, since thedevice 3400 has rotated ninety degrees in a clockwise direction (bygoing from being held vertically to being held horizontally) therotation operation would cause the display areas 1180 and 1155 to rotateninety degrees in a counter clockwise direction. As the display areas1180 and 1155 rotate, the display areas 1180 and 1155 shrinkproportionally to fit the UI 1105 so that the display areas 1180 and1155 may still appear entirely on the UI 1105. Some embodiments mayprovide a message to indicate the state of this device 3400 (e.g., bydisplaying the word “Rotating”).

The fifth stage 3430 illustrates the UI 1105 after the display areas1180 and 1155 have rotated ninety degrees counter clockwise fromportrait view to landscape view. In this stage, the display areas 1180and 1155 have been rotated but have not yet expanded across the fullwidth of the UI 1105. The arrows 3475 indicate that at the end of thefifth stage, the display areas 1180 and 1155 will start to laterallyexpand to fit the full width of the UI 1105. Different embodiments maynot include this stage since the expansion could be performedsimultaneously with the rotation in the fourth stage 3425.

The sixth stage 3435 illustrates the UI 1105 after the display areas1180 and 1155 have been expanded to occupy the full display of the UI1105. As mentioned above, other embodiments may implement this rotationdifferently. For some embodiments, simply rotating the screen of adevice past a threshold amount may trigger the rotation operation,notwithstanding the orientation of the device 3400.

Also, other embodiments might provide a different animation forindicating the rotation operation. The rotation operation performed inFIG. 34 involves the display areas 1180 and 1155 rotating about thecenter of the UI 1105. Alternatively, the display areas may beindividually rotated about the center axis of their individual displayareas. One such approach is shown in FIG. 35 . FIG. 35 shows analternative method to animating the rotation of the display areas 1170and 1160 of PIP display 1180 of a UI 1105. The PIP display 1180illustrated in FIG. 35 is the same PIP display 1180 illustrated in FIG.11 .

FIG. 35 illustrates the rotation of the PIP display 1180 in terms of sixdifferent operational stages 3410, 3415, 3420, 3525, 3530, and 3535. Thefirst three stages of operation of the UI 1105 are identical to thefirst three stages of operation as described in the UI 1105 in FIG. 34 .At the third stage for both FIGS. 34 and 35 , the device 3500 has gonefrom being held vertically to being held horizontally and the rotationof the UI 1105 has not yet begun.

The fourth stage 3525 illustrates the alternative method to animatingthe rotation. In this stage, the rotation operation has started.Specifically, the fourth stage shows 3525 the start of the rotation ofthe display areas 1170 and 1160. The display areas 1170 and 1160 eachrotate around axes 3567 and 3565, respectively, going through the centerof each of the display areas (i.e., the z-axis). The display areas 1170and 1160 are rotated the same amount but in the opposite direction ofthe rotation of the device 3500 (e.g., through the tilting of the device3500). Similar to that illustrated in the fourth stage 3425 of FIG. 34above, since the device 3500 has rotated ninety degrees in a clockwisedirection (by going from being held vertically to being heldhorizontally) the rotation operation would cause the display areas 1170and 1160 to rotate ninety degrees in a counter clockwise direction. Asthe display areas 1170 and 1160 rotate, the display areas 1170 and 1160shrink proportionally to fit the UI 1105 so that the display areas 1170and 1160 may still appear entirely on the UI 1105.

The fifth stage 3530 illustrates the UI 1105 after each of the displayareas 1170 and 1160 have rotated ninety degrees counter clockwise fromportrait view to landscape view. In this stage, the display areas 1170and 1160 have been rotated but have not yet expanded across the fullwidth of the UI 1105. Moreover, the display area 1160 has not moved intoits final position. The final position of the inset display area 1160 inthe PIP display 1180 is determined by the position of the inset displayarea 1160 in the PIP display 1180 as shown in the first stage 3410(e.g., the inset display area 1160 in the lower left corner of the PIPdisplay 1180). In this stage, the inset display area 1160 is still inthe upper left corner of the UI 1105.

The arrows 3580 indicate that at the end of the fifth stage 3530, thedisplay areas 1170 and 1160 will start to laterally expand until themain display area 1170 fits the full width of the UI 1105 for a devicethat is held horizontally. Moreover, the arrow 3575 indicates that theinset display area 1160 will slide to the lower left corner of the PIPdisplay 1180.

Different embodiments may implement this differently. In someembodiments, the moving of the inset display area 1160 may occursimultaneously as the expansion of the main display area 1170 orsequentially. Moreover, some embodiments may resize the inset displayareas 1160 before, during or after the expansion of the main displayarea 1170 to create the new PIP display 1180. In this example, thedisplay area 1155 disappears while the display areas 1160 and 1170 arerotating. However, the display area 1155 may remain on the UI 1105during the rotation and rotate along with the display areas 1160 and1170 in some embodiments.

The sixth stage 3535 illustrates the UI 1105 after the inset displayarea 1160 has reached its new location and the display areas 1160 and1170 have been properly expanded to fit the full width of the UI 1105.In this example, the inset display area 1160 is now in the lower leftcorner of the PIP display 1180, overlapping the main display area 1170.The PIP display 1180 now has the same display arrangement as the PIPdisplay 1180 from the first stage 3410. The appearance of the displayarea 1155 below the PIP display 1180 in the sixth stage indicates thatthe rotation operation is completed. As noted above, simply rotating thescreen of a device past a threshold amount may trigger the rotationoperation, notwithstanding the orientation of the device 3500.

In the examples described above by reference to FIGS. 34 and 35 , theorientation of the display area 1170 also changes (i.e., from portraitto landscape). That is, after the display area 1170 is rotated in thethird stage 3420, the orientation of the display area 1170 changes fromportrait to landscape by horizontally expanding the PIP display 1180 sothat it fills the entire UI 1105. In some embodiments, when the device3500 is rotated, video captured by the remote device rotates but theorientation of the display area that displays the video captured by theremote device remains unchanged. One such example is illustrated in FIG.36 . This figure is similar to FIG. 35 except that video displayed inthe display area 1170 rotates but the display area 1170 remainsdisplayed in portrait orientation.

FIG. 36 also illustrates an example of a rotation operation in which thedisplay area 1155 remains in the same position (instead of rotating andexpanding horizontally to fill the PIP display 1180 as shown in FIG. 35). Moreover, this figure includes a layout of the display area 1155 thatis the same as the layout of the display area 1155, described above inFIG. 12 . As shown, the display area 1155 remains in the same positionas the device 3500 rotates in the stages 3640, 3645, 3650, 3655, 3685,and 3690.

Some embodiments provide a rotation operation in which the orientationof the display area that displays video captured by the local devicechanges (instead of remaining in the same orientation as shown in FIG.35 ) to reflect the orientation of the local device after the rotationoperation is performed on the local device. FIG. 36 illustrates anexample of such a rotation operation of a UI 1105 by reference to sixdifferent stages 3640, 3645, 3650, 3655, 3685, and 3690. In this figure,the first stage 3640 shows the inset display area 1160, which displaysvideo captured by a camera of the device 3500, in a portraitorientation. The second and third stages 3645 and 3650 are similar tothe second and third stages 3415 and 3420 of FIG. 35 as they show thetilting of the device 3500 at various stages of the rotation operation.At this point, the camera of the device 3500 is capturing images in alandscape orientation. To indicate this transition, some embodimentsprovide an animation as shown in fourth and fifth stages 3655 and 3685while other embodiments do not provide any animation at all.

In the fourth stage 3655, the image displayed in the inset display area1160 is rotated, but not the inset display area 1160 itself since thetilting of the device 3500 in the second and third stages 3445 and 3650has rotated the inset display area 1160 to a landscape orientation. Inthe fifth stage 3685, the rotated image in the inset display area 1160is horizontally expanded to fill the inset display area 1160 and theinset display area 1160 starts to move towards the lower left area ofthe PIP display 1180 to position the inset display area 1160 in the samerelative position as the inset display area 1160 in the PIP display ofthe first stage 3640.

In some embodiments, the orientation of the display area that displaysthe video captured by the remote device also changes to reflect theorientation of the remote device after a rotation operation is performedon the remote device. FIG. 37 illustrates four different stages of a UI1105 of the device 3500 in which (1) the orientation of the display areathat displays the video captured by the local device (display area 1160in this example) changes to reflect the orientation of the local deviceafter a rotation operation is performed on the local device and (2) theorientation of the display area that displays video captured by theremote device (display area 1170 in this example) changes to reflect theorientation of the remote device after a rotation operation is performedon the remote device.

In the first stage 3705, the UI 1105 is the same as the UI 1105 in FIG.36 . Specifically, the first stage 3705 shows the display areas 1160 and1170 in a portrait orientation because the device 3500 is shown in aportrait orientation and the remote device is in a portrait orientation(not shown). From the first stage 3705 to the second stage 3710, arotation operation is performed on the local device by rotating thedevice 3500 ninety degrees from an upright position to a sidewaysposition. The second stage 3710 shows the UI 1105 after the rotationoperation of the device 3500 is completed. In this stage, the videosdisplayed in the display areas 1170 and 1160 have rotated to an uprightposition. However, only the display area 1160 of the locally capturedvideo has rotated from a portrait orientation to a landscape orientationsince the rotation operation is only performed on the local device(i.e., the device 3500). The display area 1170 remains in the portraitorientation.

From the second stage 3710 to the third stage 3715, a rotation operationis performed on the remote device by rotating the remote device from anupright position to a sideways position (not shown). The third stage3715 shows the UI 1105 after the rotation operation of the remote deviceis completed. In this stage, the video displayed in the display area1170 and the display area 170 of the remotely captured video haverotated from a portrait orientation to a landscape orientation since therotation operation is only performed on the remote device. Thus, thisstage of the UI 1105 displays the display areas 1170 and 1160 of thelocally and remotely captured videos both in landscape orientation.

From the third stage 3715 to the fourth stage 3720, a rotation operationis performed on the local device by rotating the device 3500 ninetydegrees from a sideways position to an upright position. The fourthstage 3720 shows the UI 1105 after the completion of this rotationoperation. In this fourth stage 3720, the videos displayed in thedisplay areas 1160 and 1170 have rotated to an upright position.However, only the display area 1160 of the locally captured video hasrotated from a landscape orientation to a portrait orientation since therotation operation is only performed on the local device (i.e., thedevice 3500). The display area 1170 remains in the landscapeorientation.

From the fourth stage 3720 to the first stage 3705, a rotation operationis performed on the remote device by rotating the remote device ninetydegrees from a sideways position to an upright position (not shown). Inthis case, the first stage 3705 shows the display area 1170 after thecompletion of this rotation operation. Therefore, the UI 1105 of thisstage shows the display areas 1160 and 1170 in a portrait orientation.Although FIG. 37 illustrates a sequence of different rotationoperations, other embodiments can perform any number of rotationoperations in any number of different sequences.

FIGS. 34, 35, 36, and 37 describe rotate operations performed on localand remote devices during a video conference. When a rotate operation isperformed on the local mobile device, some embodiments notify the remotedevice of the rotate operation in order for the remote device to performany modifications to the local device's video (such as rotating thedisplay area that is displaying the local device's video). Similarly,when a rotate operation is performed on the remote device, the remotedevice notifies the local device of this operation to allow the localdevice to perform any modifications the remote device's video. Someembodiments provide a control communication channel for communicatingthe notification of rotate operations between the local and remotedevices during the video conference.

Even though FIGS. 34, 35, 36, and 37 illustrate different manners inwhich the animation of a rotation can be performed, one of ordinaryskill will realize that other embodiments may display the animation ofthe rotation in other different ways. In addition, the animation of therotation operation can cause changes to the image processing operationsof the local mobile device such as causing the video conference manager1604 to re-composite the display area(s) at different angles in the UI1105 and scale the images displayed in the display area(s).

3. Window Size Adjustment

Some embodiments allow a user of a mobile device to adjust the size ofan inset display area of a PIP display presented during a videoconference. Different embodiments provide different techniques forresizing an inset display area. FIG. 38 illustrates one approach forresizing the inset display area. In this approach, the user of themobile device adjusts the size of the inset display area by selecting acorner of the inset display area and then expanding or shrinking theinset display area.

In FIG. 38 , a UI 3800 of a mobile device 3825 presents a PIP display3865 during a video conference with a remote user of another mobiledevice. This PIP display 3865 includes two video displays: a backgroundmain display area 3830 and a foreground inset display area 3835. Thebackground main display area 3830 takes up a majority of the PIP display3865 while the foreground inset display area 3835 is smaller andoverlaps the background main display area 3830. In this example, thebackground main display area 3830 presents a video of a person holding aguitar, which is assumed to be a person whose video is being captured bythe remote device's front camera or a person whose video is beingcaptured by the remote device's back camera. The foreground insetdisplay area 3835 presents a video of a person with a hat, which, inthis example, is assumed to be a person whose video is being captured bythe local device's front camera or a person whose video is beingcaptured by the local device's back camera. Below the PIP display 3865is a display area 1155 that includes a selectable UI item 3860 labeled“End Conference” (e.g. a button 3860) that allows the user to end thevideo conference by selecting the item.

This PIP display 3865 is only one manner of presenting a composite viewof the videos being captured by the remote and local devices. Someembodiments may provide other composite views. For instance, instead ofhaving a larger background display for the video from the remote device,the larger background display can be of the video from the local deviceand the smaller foreground inset display can be of the video from theremote device. Also, some embodiments allow the local and remote videosto appear in the UI 3800 in two side-by-side display areas (e.g. leftand right display windows, or top and bottom display windows) or twodiagonally aligned display areas. The manner of the PIP display or adefault display mode may be specified by the user in some embodiments.In other embodiments, the PIP display may also contain a largerbackground display and two smaller foreground inset displays.

FIG. 38 illustrates the resize operation in terms of four operationalstages of the UI 3800. In the first stage 3805, the foreground insetdisplay 3835 is substantially smaller than the background main displayarea 3830. Also in this example, the foreground inset display area 3835is located at the lower right corner of the PIP display 3865. In otherexamples, the foreground inset display area 3835 may be a different sizeor located in a different area in the PIP display 3865.

In the second stage 3810, the resizing operation is initiated. In thisexample, the operation is initiated by selecting a corner of the insetdisplay area 3835 that the user wants to resize (e.g., by holding afinger 3840 down on the upper left corner of the inset display area3835). The second stage 3810 of the UI 3800 indicates this selection interms of a thick border 3845 for the inset display area 3835. At thisstage, the user can expand or shrink the inset display area 3835 (e.g.,by dragging his finger 3840 on the PIP display 3865 away from the insetdisplay area 3835 or toward the inset display area 3835).

The third stage 3815 illustrates the UI 3800 after the user has startedto expand the inset display area 3835 by moving his finger 3840 awayfrom the inset display area 3835 (i.e., by moving his finger diagonallytowards the upper left corner of the UI 3800 in this example), asindicated by an arrow 3850. Also as indicated by arrow 3855, themovement of the finger 3840 has expanded the inset display area 3835proportionally in both height and width. In other examples, the user canshrink the inset display area 3835 using the same technique (i.e., bydragging the finger toward the inset display area 3835).

The fourth stage 3820 displays the UI 3800 after the resizing of theinset display area 3835 has been completed. In this example, the usercompletes the resize of the inset display area 3835 by stopping thedragging of his finger 3840 and removing his finger from the PIP display3865 once the inset display area 3835 has reached the desired size. As aresult of this procedure, the resized inset display area 3835 is largerthan its original size in the first stage 3805. The removal of the thickborder 3845 indicates that the inset display area resize operation isnow completed.

Some embodiments provide other techniques for allowing a user to resizean inset display area 3835 in a PIP display 3865 during a videoconference. FIG. 39 illustrates one such other technique. This figureillustrates a technique for resizing the inset display area 3835 byselecting an edge of the inset display area 3835 (i.e., on one of thesides of this display area 3835) and then expanding or shrinking insetdisplay area 3835.

FIG. 39 illustrates this resizing operation in terms of four operationalstages of the UI 3800 of FIG. 38 . The first stage 3805 in FIG. 39 isthe same as the first stage 3805 in FIG. 38 . Specifically, in thisstage, the UI 3800 of device 3925 illustrates the PIP display 3865 witha larger background main display area 3830 and a smaller foregroundinset display area 3835 at the bottom right corner of the PIP display3865. Even though FIGS. 38 and 39 illustrate two different techniquesfor resizing an inset display area 3835 in the same UI 3800, one ofordinary skill will realize that some embodiments will not offer boththese techniques in the same UI.

The second stage 3910 illustrates the start of a resizing operation. Inthis example, the user initiates the operation by selecting a side ofthe inset display area 3835 that the user wants to resize (e.g., byplacing a finger 3840 down on the top edge or the side edge of the insetdisplay area 3835). In this example, a user places his finger 3840 onthe top edge of the inset display area 3835 in order to make thisselection. The second stage 3910 indicates this selection in terms of athick border 3845 for the inset display area 3835.

The third stage 3915 illustrates the UI 3800 after the user has startedto expand the inset display area 3835 by moving his finger 3840 awayfrom the inset display area 3835 (i.e., vertically toward the top of thePIP display 3865), as indicated by an arrow 3950. Also as indicated byarrow 3955, the movement of the finger 3840 has expanded the insetdisplay area 3835 proportionally in both height and width. In otherexamples, the user can shrink the display area 3835 using the sametechnique (e.g., by dragging the finger 3840 toward the inset displayarea 3835).

The fourth stage 3920 displays the UI 3800 after the resizing of theinset display area 3835 has been completed. In this example, the usercompletes the resize of the inset display area 3835 by stopping thedragging of his finger 3840 and removing his finger 3840 from thedevice's display screen once the inset display area 3835 has reached thedesired size. As a result of this procedure, the resized inset displayarea 3835 is larger than its original size in the first stage 3805. Theremoval of the thick border 3845 indicates that the inset display arearesize operation is now completed.

In response to a drag operation, some embodiments adjust the size of theinset display area 3835 proportionally in height and width, asillustrated by FIGS. 38 and 39 . Other embodiments may allow the user toadjust the height and/or width of an inset display area 3835 withoutaffecting the other attribute. FIG. 40 illustrates an example of onesuch resizing approach.

Specifically, FIG. 40 illustrates a UI 3800 of a mobile device 4025 thatis similar to the UI 3800 of FIG. 38 except the UI 3800 of FIG. 40allows the inset display area 3835 to be expanded in the horizontaldirection and/or vertical direction when one of the edges of the insetdisplay area 3835 is selected and moved horizontally or vertically. Tosimplify the description of the UI 3800, FIG. 40 illustrates a PIPdisplay 3865 in the UI 3800 that is similar to the PIP display 3865 ofFIG. 38 except now the inset display area 3835 is in the upper rightcorner of the PIP display 3865. The PIP display 3865 includes two videodisplays: a background main display area 3830 and a foreground insetdisplay area 3835. In this example, the background main display area3830 presents a video that is being captured by the remote device'sfront camera or back camera. The foreground inset display area 3835presents a video that is being captured by the local device's frontcamera or back camera.

Like FIG. 38 , FIG. 40 illustrates the resizing operation in terms offour operational stages of the UI 3800. The first stage 4005 is similarto the first stage 3805 of FIG. 38 except now the inset display area3835 is in the upper right corner. The other three stages 4010, 4015 and4020 are similar to the three stages 3910, 3915 and 3920 except that theselection and movement of the bottom edge of the inset display area 3835has caused the inset display area 3835 to only expand in the verticaldirection without affecting the width of the inset display area 3835.

FIGS. 38, 39, and 40 provide examples embodiments that allow the user toresize an inset display area 3835 of a PIP display 3865 by selecting acorner or a side of the inset display area 3835. Some embodimentsprovide other techniques for resizing an inset window 3835. Forinstance, FIG. 41 illustrates that some embodiments allow the insetdisplay area 3835 to be resized by selecting the interior of the insetdisplay area 3835. In this approach, the user adjusts the size of theinset display area 3835 by placing two fingers 4155 and 4156 on thescreen and dragging the fingers either away from or toward each other.

In FIG. 41 , a UI 3800 of a mobile device 4140 provides a PIP display3865 during a video conference with a remote user of another mobiledevice. To simplify the description of the UI 3800, FIG. 41 illustratesa PIP display 3865 in this UI 3800 that is similar to the PIP display3865 of FIG. 38 .

FIG. 41 illustrates the resizing operation in terms of seven operationalstages of the UI 3800. The first four stages 3805, 4110, 4115, and 4120show the expansion of an inset display area 3835 while the last threestages show the shrinking of the inset display area 3835. The firststage 3805 in FIG. 41 is the same as the first stage 3805 in FIG. 38 .Specifically, in this stage, the UI 3800 illustrates the PIP display3865 with a larger background main display area 3830 and a smallerforeground inset display area 3835. In this example, the background maindisplay area 3830 presents a video that is being captured by the remotedevice's front camera or back camera. The foreground inset display area3835 presents a video that is being captured by the local device's frontcamera or back camera.

The second stage 4110 illustrates the UI 3800 after the resizingoperation is initiated. In this example, the user initiates theoperation by selecting the inset display area 3835 that the user wantsto resize (e.g., by placing two fingers 4155 and 4156 down within theinset display area 3835). The second stage 4110 of the UI 3800 indicatesthis selection in terms of a thick border 4190 for the inset displayarea 3835.

The third stage 4115 illustrates the UI 3800 after the user has startedto expand the inset display area 3835 by moving his fingers 4155 and4156 away from each other (i.e., moving finger 4155 toward the upperleft corner of the PIP display 3865 and moving finger 4156 toward thelower right corner of the PIP display 3865), as indicated by arrows4160. As indicated by an arrow 4165, the movement of the fingers 4155and 4156 has expanded the inset display area 3835 proportionally in bothheight and width.

The fourth stage 4120 displays the UI 3800 after the resizing of theinset display area 3835 has been completed. In this example, the usercompletes the resize of the inset display area 3835 by stopping thedragging of his fingers 4155 and 4156 and removing his fingers 4155 and4156 from the device's display screen. As a result of this procedure,the resized inset display area 3835 is larger than its original size inthe first stage 3805. The removal of the thick border 4190 indicatesthat the inset display area resize operation is now completed.

In the fifth stage 4125, the user re-selects the inset display area 3835by placing down two fingers 4155 and 4156 on the inset display area3835. The sixth stage 4130 illustrates the UI 3800 after the user hasstarted to shrink the inset display area 3835 by moving his fingers 4155and 4156 closer together, as indicated by arrows 4170. As indicated byan arrow 4175, the movement of the fingers 4155 and 4156 has shrunk theinset display 3835 proportionally in both height and width.

The seventh stage 4135 is similar to the fourth stage 4120 in FIG. 41 ,except that the inset display area 3835 has shrunk in size as a resultof the operation. The removal of the thick border 4190 indicates thatthe inset display area resize operation is now completed.

The above description of FIGS. 38-41 illustrates several example userinterfaces that allow a user to resize an inset display area of a PIPdisplay. In some embodiments, the resizing of an inset display areacauses changes to the image processing operations of the dual cameramobile device such causing the video conference manager 1604 to changethe scaling and compositing of the inset display area in the PIP displayin response to the user's input. In addition, in some embodiments thelayout of the display area 1155 in FIGS. 38-41 is the same as the layoutof the display area 1155 of FIG. 12 , described above.

4. Identifying Regions of Interest

Some embodiments allow a user to identify a region of interest (ROI) ina displayed video during a video conference in order to modify the imageprocessing (e.g., the image processing manager 1608 in FIG. 16 ), theencoding (e.g., the encoder 1655 in FIG. 16 ), the behavior of themobile devices and their cameras during the video conference, or acombination thereof. Different embodiments provide different techniquesfor identifying such a region of interest in a video. FIG. 42illustrates a user interface of some embodiments for identifying aregion of interest in a video in order to improve the image quality ofthe video.

In FIG. 42 , a UI 4200 of a mobile device 4225 presents a PIP display4265 during a video conference with a remote user of another mobiledevice. The PIP display in FIG. 42 is substantially similar to the onein FIG. 41 . Specifically, the PIP display in FIG. 42 includes two videodisplays: a background main display 4230 and a foreground inset display4235. In this example, the background main display 4230 presents a videoof a tree and a person with a hat, which are assumed to be a tree and aperson whose video is being captured by the remote device's front cameraor a tree and a person whose video is being captured by the remotedevice's back camera. The foreground inset display 4235 presents a videoof a man, which in this example is assumed to be a man whose video isbeing captured by the local device's front camera or a person whosevideo is being captured by the local device's back camera. Below the PIPdisplay is a display area 1155 that includes a selectable UI item 4260labeled “End Conference” (e.g. a button 4260) that allows the user toend the video conference by selecting the item.

This PIP display is only one manner of presenting a composite view ofthe videos being captured by the remote and local devices. Someembodiments may provide other composite views. For instance, instead ofhaving a larger background display for the video from the remote device,the larger background display can be of the video from the local deviceand the smaller foreground inset display can be of the video from theremote device. Also, some embodiments allow the local and remote videosto appear in the UI in two side-by-side display areas (e.g. left andright display windows, or top and bottom display windows) or twodiagonally aligned display areas. In other embodiments, the PIP displaymay also contain a larger background display and two smaller foregroundinset displays. The manner of the PIP display or a default display modemay be specified by the user in some embodiments.

FIG. 42 illustrates the ROI identification operation in terms of fouroperational stages of the UI 4200. As shown in the first stage 4205, thevideo presented in the background display 4230 has very low quality(i.e., the video images are fuzzy). In this example, a user of a mobiledevice 4225 would like to identify the area in the background display4230 where the person's face 4270 appears as the region of interest.

In the second stage 4210, the operation of identifying a region ofinterest is initiated. In this example, the operation is initiated byselecting an area in the video presented in the background display 4230that the user wants to identify as the region of interest (e.g., bytapping a finger 4250 on the device's screen at a location about thedisplayed person's face 4270 in the background display 4230).

As shown in the third stage 4215, the user's selection of the areacauses the UI 4200 to draw an enclosure 4275 (e.g., a dotted square4275) surrounding the area of the user's selection. The fourth stage4220 displays the UI 4200 after the identification of the region ofinterest has been completed. As a result of this process, the quality ofthe video within the region of interest has been substantially improvedfrom that in the first stage 4205. The removal of the enclosure 4275indicates that the ROI selection operation is now completed. In someembodiments, the ROI identification process also causes the same changesto the same video displayed on the remote device as it does to the localdevice 4225. In this example for instance, the picture quality withinthe region of interest of the same video displayed on the remote deviceis also substantially improved.

In some embodiments, the user may enlarge or shrink the enclosure 4275in the third stage 4215 (e.g., by holding the finger 4250 down on thedisplay and moving the finger 4250 toward the upper right corner of thescreen to enlarge the enclosure 4275 or moving the finger 4250 towardthe lower left corner of the screen to shrink the enclosure 4275). Someembodiments also allow the user to relocate the enclosure 4275 in thethird stage 4215 (e.g., by holding the finger 4250 down on the displayand moving the finger 4250 horizontally or vertically on the display).In some other embodiments, the selection of the area may not cause theUI 4200 to draw the enclosure 4275 at all in the third stage 4215.

Other embodiments provide different techniques for allowing a user toidentify a region of interest in a video. FIG. 43 illustrates one suchother technique. In FIG. 43 , the user identifies a region of interestby drawing a shape that bounds the region. The shape in this example isa rectangle, but it can be other shapes (e.g., any other polygon, acircle, an ellipse, etc.). Some embodiments provide this alternativetechnique of FIG. 43 in a device UI that also provides the techniqueillustrated in FIG. 42 . Other embodiments, however, do not provide boththese techniques in the same UI.

FIG. 43 illustrates this ROI identification operation in terms of fiveoperational stages of a UI 4200. The first stage 4205 in FIG. 43 isidentical to the first stage 4205 in FIG. 42 . Specifically, in thisfirst stage 4205, the UI 4200 illustrates a PIP display 4265 with alarger background main display 4230 and a smaller foreground insetdisplay 4235 at the bottom left corner of the PIP display 4265.

In the second stage 4310, the operation of identifying a region ofinterest is initiated. In this example, the operation is initiated byselecting for a duration of time a first position for defining theregion of interest in a video presented in the background display area4230 (e.g., by holding a finger 4350 down on the device's screen at alocation about the displayed person's face 4270 in the backgrounddisplay 4230 for a duration of time). In the third stage 4315, the UI4200 indicates that the first position 4370 has been selected in termsof a dot 4355 next to the selected first position on the backgrounddisplay area 4230.

The fourth stage 4320 illustrates the UI 4200 after the user hasselected a second position 4375 for defining the region of interest. Inthis example, the user selects this second position 4375 by dragging thefinger 4350 across the device's screen from the first location after thedot 4355 appears and stopping at a location between the displayed hatand the displayed tree in the background display area 4230, as indicatedby an arrow 4360. As shown in the fourth stage, this dragging caused theUI 4200 to draw a rectangular border 4365 for the region of interestarea that has the first and second positions 4370 and 4375 at itsopposing vertices.

The fifth stage 4325 illustrates the UI 4200 after identification of theregion of interest has been completed. In this example, the usercompletes identification of the region of interest by stopping thedragging of the finger 4350 and removing the finger 4350 from thedevice's display screen once the desired region of interest area hasbeen identified. The fifth stage 4325 illustrates that as a result ofthe drawing process, the quality of the video within the region ofinterest has been substantially improved from that in the first stage4205. In some embodiments, the drawing process also causes the samechanges to the display on the remote device as it does to the localdevice 4225. In this example for instance, the picture quality withinthe region of interest of the same video displayed on the remote devicewill be substantially improved.

The description of FIGS. 42 and 43 , above, illustrates differentmanners of identifying a region of interest in a video in order toimprove the picture quality of the identified region. In someembodiments, improving the picture quality of the identified region ofinterest causes changes to the encoding operations of the dual cameramobile device such as allocating more bits to the identified region whenencoding the video.

Some embodiments allow the user to identify a region of interest in avideo to make different changes to the mobile devices or their cameras.For instance, FIG. 44 illustrates an example of identifying a region ofinterest in a video to expand or shrink the region of interest area onthe display. In this approach, the user identifies a region of interestin a video by selecting an area on the display as the center of theregion of interest and then expanding or shrinking the region ofinterest area.

In FIG. 44 , a UI 4400 of a mobile device 4425 presents a PIP display4265 during a video conference with a remote user of another mobiledevice. The PIP display 4265 in FIG. 44 is substantially similar to thePIP display 4265 of FIG. 42 , but the foreground inset display 4235 ofFIG. 44 is located in the lower left corner of the PIP display 4265.

FIG. 44 illustrates the ROI selection operation in terms of fouroperational stages of the UI 4400. As shown in the first stage 4405, thebackground display 4430 presents a video with a man on the left and atree 4440 on the right of the display 4430. Moreover, the tree 4440 isrelatively small and occupies only the right side of the backgrounddisplay area 4430. In this example, a user of a mobile device 4425 wouldlike to identify the area where the tree 4440 appears on the display4430 as the region of interest.

In the second stage 4410, the operation of identifying a region ofinterest is initiated. In this example, the operation is initiated byselecting an area 4440 in the video presented in the background display4430 that the user wants to identify as the region of interest (e.g., byholding two fingers 4445 and 4446 down on the background display area4430 where the tree 4440 is displayed). At this stage 4410, the user canmake the region of interest area 4440 expand and take a larger portionof the background display area 4430 by dragging his fingers 4445 and4446 farther away from each other. The user can also make the region ofinterest 4440 shrink to take a smaller portion of the background displayarea 4430 by dragging his fingers 4445 and 4446 closer together.

The third stage 4415 illustrates the UI 4400 after the user has startedto make the region of interest 4440 expand to take up a larger portionof the background display area 4430 by moving his fingers 4445 and 4446farther away from each other (i.e., the finger 4445 moves toward theupper left corner of the background display area 4430 and the finger4446 moves toward the lower right corner of the display 4430), asindicated by arrows 4450. In some embodiments, the finger movement alsocauses the same changes to the display of the remote device as it doesto the local device. In this example for instance, the region ofinterest of the same video will expand and take up a larger portion ofthe background display area 4430 of the remote device. In someembodiments, the expansion of the region of interest in the localdisplay and/or remote display causes one or both of the mobile devicesor their cameras to modify one or more of their other operations, asfurther described below.

The fourth stage 4420 displays the UI 4400 after the identification ofthe region of interest has been completed. In this example, the usercompletes the identification of the region of interest by stopping thedragging of his fingers 4445 and 4446 and removing the fingers 4445 and4446 from the device's display screen once the region of interest hasreached the desired proportion in the background display area 4430. As aresult of this process, the region of interest has taken up a majorityof the background display 4430. The identification of the region ofinterest operation is now completed.

Some of the examples above illustrate how a user may identify a regionof interest in a video for improving the image quality within theselected region of interest in the video (e.g., by increasing the bitrate for encoding the region of interest portion of the video). In someembodiments, identifying a region of interest in the video causeschanges to the image processing operations of the mobile device such asexposure, scaling, focus, etc. For example, identifying a region ofinterest in the video can cause the video conferencing manager 1604 toscale and composite the images of the video differently (e.g.,identifying a region of interest to which to zoom).

In other embodiments, identifying a region of interest in the videocauses changes to the operation of the mobile device's camera(s) (e.g.,frame rate, zoom, exposure, scaling, focus, etc.). In yet otherembodiments, identifying a region of interest in the video causeschanges to the encoding operations of the mobile device like allocatingmore bits to the identified region, scaling, etc. In addition, while theexample ROI identification operations described above may cause only oneof the above-described modifications to the mobile device or itscameras, in some other embodiments the ROI identification operation maycause more than one of the modifications to the operation of the mobiledevice or its cameras. In addition, in some embodiments, the layout ofthe display area 1155 in FIGS. 42-44 is the same as the layout of thedisplay area 1155 of FIG. 12 , described above.

B. Switch Camera

Some embodiments provide procedures to switch cameras (i.e., change thecamera by which images are captured) during a video conference.Different embodiments provide different procedures for performing theswitch camera operation. Some embodiments provide procedures performedby a dual camera mobile device for switching cameras of the device(i.e., local switch) while other embodiments provide procedures for thedual camera mobile device to instruct another dual camera mobile devicein the video conference to switch cameras of the other device (i.e.,remote switch). Yet other embodiments provide procedures for both.Section IV.B.1 will describe a process for performing a local switchcamera operation on a dual camera mobile device. Section IV.B.2 willdescribe a process for performing a remote switch camera operation onthe dual camera mobile device.

1. Local Switch Camera

FIG. 45 illustrates a process 4500 that some embodiments perform on alocal dual camera mobile device to switch between the two cameras of thedevice during a video conference with a remote mobile device thatincludes at least one camera. In some embodiments, the process 4500 isperformed by the video conference manager 1604 shown in FIG. 16 . Forpurposes of explanation, the discussion will refer to one camera of thelocal dual camera mobile device as camera 1 and the other camera of thelocal dual camera mobile device as camera 2.

The process 4500 begins by starting (at 4505) a video conference betweenthe local dual camera mobile device and the remote mobile device. Next,the process 4500 sends (at 4510) a video image from the currentlyselected camera (e.g., the camera 1) of the local dual camera mobiledevice to the remote mobile device for display on the remote mobiledevice. At 4510, the process also generates and displays a compositedisplay based on this video image and the video image that it receivesfrom the remote mobile device.

The process 4500 then determines (at 4515) whether a request to end thevideo conference is received. As described above, a video conference canend in some embodiments at the request of a user of the local dualcamera mobile device (e.g., through a user interface of the local dualcamera mobile device) or a user of the remote mobile device (e.g.,through a user interface of the remote mobile device). When the process4500 receives a request to end the video conference, the process 4500ends.

When the process 4500 does not receive a request to end the videoconference, the process 4500 then determines (at 4520) whether the userof the local dual camera mobile device has directed the device to switchcameras for the video conference. The process 4500 returns to operation4510 when the process 4500 determines (at 4520) that it has not beendirected to switch cameras. However, when the process 4500 determines(at 4520) that it has been so directed, the process 4500 transitions to4525.

At 4525, the process 4500 sends a notification to the remote mobiledevice to indicate that the local dual camera mobile device is switchingcameras. In some embodiments, the process 4500 sends the notificationthrough the video conference control channel that is multiplexed withthe audio and video channels by the VTP Manager 3125 as described above.

After sending its notification, the process 4500 performs (at 4530) aswitch camera operation. In some embodiments, performing (at 4530) theswitch camera operation includes instructing the CIPU to stop capturingvideo images with the camera 1 and to start capturing video images withthe camera 2. These instructions can simply direct the CIPU to switchcapturing images from the pixel array associated with the camera 2 andto start processing these images. Alternatively, in some embodiments,the instructions to the CIPU are accompanied by a set of initializationparameters that direct the CIPU (1) to operate the camera 2 based on aparticular set of settings, (2) to capture video generated by the camera2 at a particular frame rate, and/or (3) to process video images fromthe camera 2 based on a particular set of settings (e.g., resolution,etc.).

In some embodiments, the switch camera instruction (at 4530) alsoincludes instructions for switching the unused camera to the fourthoperational power mode as described above. In this example, the switchcamera instructions include instructions for the camera 2 to switch toits fourth operational power mode. In addition, the switch camerainstructions also include instructions for the camera 1 to switch fromits fourth operational power mode to another operational power mode suchas the first operational power mode to conserve power or to the thirdoperational power mode so it can quickly switch to the fourthoperational power mode and start capturing images when requested to doso. The switch camera operation 4530 also involves compositing imagescaptured by the camera 2 of the local dual camera mobile device (insteadof images captured by the camera 1) with images received from the remotemobile device for display on the local dual camera mobile device.

After directing the switch camera at 4530, the process 4500 performs (at4535) a switch camera animation on the local dual camera mobile deviceto display a transition between the display of images from the camera 1and the display of images from the camera 2. Following the switch cameraanimation on the local dual camera mobile device, the process 4500 loopsback through operations 4510-4520 until an end video conference requestor a new switch camera request is received.

FIG. 46 illustrates one example of how some embodiments allow a switchcamera operation to be requested through a UI 1105 of a dual cameradevice and how these embodiments animate the switch camera operation.This figure illustrates the switch camera operation in terms of eightdifferent operational stages 4610, 4615, 4620, 4625, 4630, 4635, 4640,and 4645 of the UI 1105 of the device. The first four stages 4610, 4615,4620, and 4625 of the UI 1105 illustrate an example of receiving auser's request to switch cameras. The user of the device has othermechanisms to make such a request in some embodiments of the invention.

The first stage 4610 is the same as the fifth stage 1130 of the UI 1105of FIG. 11 , which shows the UI 1105 after a video conference is set up.At this stage, the UI 1105 displays a PIP display that includes twovideo displays: a larger background display from the remote camera and asmaller foreground inset display from the local camera. In this example,the background main display area 1170 presents a video of a lady, whichin this example is assumed to be a lady whose video is being captured bythe remote device, while the foreground inset display area 1160 presentsa video of a man, which in this example is assumed to be a man whosevideo is being captured by the local device's front camera.

The second stage 4615 then shows the initiation of the switch cameraoperation through the selection of the PIP display area 1180 of the UI1105. As shown, a selection is made by placing the user's finger 4670 onthe PIP display 1180. The third stage 4620 shows the UI 1105 thatincludes a selectable UI item 4675 (e.g., switch camera button 4675) forrequesting a switch between the cameras of the local device 4600 duringthe video conference. The fourth stage 4625 illustrates the UI 1105after the user of the local device 4600 selects (e.g., through a singlefinger tap) the selectable UI item 4675, and after this selection isindicated through the highlighting of the selectable UI item 4675. Byselecting this selectable UI item 4675, the user is directing the device4600 to switch from the front camera of the device 4600 to the backcamera of the device 4600 during the video conference. In other exampleswhere the back camera of the device 4600 is capturing video, the user'sselection of the selectable UI item 4675 directs the device 4600 toswitch from the back camera of the device 4600 to the front camera ofthe device 4600. After the fourth stage, the video conference managersends instructions to the CIPU and the remote device to start the switchcamera operation.

The last four stages 4630, 4635, 4640, and 4645 of the UI 1105illustrate an example of a switch camera animation on the local device.This animation is intended to provide an impression that the videocaptured from the front and the back cameras of the local device arebeing concurrently displayed on two opposing sides of a viewing panethat can have only one of its sides viewed by the user at any giventime. When a switch camera is requested in the middle of a videoconference, this viewing pane is made to appear to rotate around thevertical axis such that the presentation of one camera's video on oneside of the viewing pane that was previously showing one camera's videoto the user rotates away from the user until it is replaced by the otherside of the viewing pane, which shows the video of the other camera.This animation and appearance of the perceived viewing pane's rotationis achieved by (1) gradually shrinking and applying perspectivecorrection operations on the video image from one camera in the displayarea for that camera, followed by (2) a gradual expansion and reductionin perspective correction operation to the video image from the othercamera in the display area.

Accordingly, the fifth stage 4630 illustrates the start of the “rotationof the viewing pane” about the vertical axis 4682. To give an appearanceof the rotation of the viewing pane, the UI 1105 has reduced the size ofthe front camera's video image in the video display area 1160, and hasapplied perspective operations to make it appear that the right side ofthe video image is farther from the user than the left side of the videoimage.

The sixth stage 4635 illustrates that the viewing pane has rotated by 90degrees such that the user can only view the edge of this pane, asrepresented by the thin line 4686 displayed in the middle of the displayarea 1160. The seventh stage 4640 illustrates that the viewing pane hascontinued to rotate such that the backside of the viewing pane 4688 isnow gradually appearing to the user in order to show the video capturedfrom the user's back camera. Again, this representation of the rotationanimation is achieved in some embodiments by reducing the size of theback camera's video image in the video display area 4688, and applyingperspective operations to make it appear that the left side of the videoimage is farther from the user than the right side of the video image.

The eighth stage 4645 illustrates the completion of the animation thatshows the switch camera operation. Specifically, this stage displays inthe display area 1160 the video image of a car that is being captured bythe back camera of the device 4600.

The example described above by reference to FIG. 46 invokes a switchcamera operation through a switch camera user interface. Otherembodiments invoke a switch camera operation differently. For example,some embodiments invoke the switch camera operation by having a switchcamera selectable UI item permanently displayed on a UI during a videoconference such the UI 1105 of FIG. 47 . In FIG. 47 , a switch camerabutton 1289 is shown in a display area 1155 along with a mute button1285 and an end conference button 1287. The layout of the display area1155 is the same layout of the display area 1155, described above byreference to FIG. 12 .

FIG. 47 illustrates the switch camera operation of a UI 1105 in terms ofsix stages: 4610, 4790, 4630, 4635, 4640, and 4645. The first stage 4610of FIG. 47 is similar to the first stage 4610 of FIG. 46 except that thelayout of the display area 1155 shows a mute button 1285, an endconference button 1287, and a switch camera button 1289 instead of asingle end conference button. The second stage 4790 illustrates the UI1105 after the user of the local device 4600 selects (e.g., through asingle finger tap using a finger 4670) the switch camera selectable UIitem 1289. In this example, by selecting this selectable UI item 1289,the user directs the device 4600 to switch from the front camera of thedevice 4600 to the back camera of the device 4600 during the videoconference. The last four stages of FIG. 47 are similar to the last fourstages of FIG. 46 except the layout of the display area 1155 is the sameas the layout described above in the first stage 4610 and therefore willnot be further described in order to not obscure the description of theinvention with unnecessary detail.

In some embodiments, when the remote mobile device receives images froma different camera of the local dual camera mobile device (i.e., thelocal dual camera mobile device switched cameras), the remote mobiledevice also performs a switch camera animation to display a transitionbetween the display of image from one camera of the local dual cameramobile device and the display of images from the other camera of thelocal dual camera mobile device. FIG. 48 illustrates an example of oneof such switch camera animation in terms of five operational stages4810, 4815, 4820, 4825, and 4830 of a UI 4805. This figure shows anexample switch camera animation on the remote mobile device 4800. Theoperational stages are the same as the example animation of FIG. 46except the animation is performed on images displayed in the displayarea 4835, which is where images from the local dual camera mobiledevice are displayed on the remote mobile device 4800. As such, theimage of the man displayed in the display area 4835 is animated toappear to rotate 180 degrees on a vertical axis 4855 located in themiddle of the display area 4850 to show the transition between thedisplay of the image of the man in the display area 4835 and the displayof the image of a car 4870. The implementation of the switch cameraanimation of some embodiments is the same as the implementation of theanimation described above.

The above example illustrates a switch camera animation on a remotedevice with a particular user interface layout. Other embodiments mightperform this switch camera animation on a remote device with a differentuser interface layout. For instance, FIG. 49 illustrates one suchexample of a remote device 4800 that has a different user interfacelayout 4805. In particular, UI 4805 of FIG. 49 has a mute button 1285,an end conference button 1287, and a switch camera button 1289 includedin a display area 1155, which is permanently displayed on one side ofthe composite display 4850 during a video conference. The layout of thethree buttons is described above by reference to FIG. 48 . Other thanthe different user interface layout, the five stages 4810, 4815, 4820,4825, and 4830 of FIG. 49 are identical to the five stages 4810, 4815,4820, 4825, and 4830 of FIG. 48 .

2. Remote Switch Camera

FIG. 50 illustrates a process 5000 for switching between two cameras ofa remote dual camera device during a video conference. This process 5000is performed by a video conference manager of a device that includes atleast one camera. In the following discussion, the device through whicha user directs a remote switch camera is referred to as the local devicewhile the device that switches between its two cameras is referred to asthe remote device. Also, in the discussion below, the remote device issaid to switch between its front camera (or camera 1) and its backcamera (or camera 2).

The process 5000 of FIG. 50 will be described by reference to FIGS. 51,52, 53, and 54 . FIG. 51 illustrates a UI 5105 of a local device 5100through which a user requests that a remote device switch between itstwo cameras during a video conference. This figure illustrates eightdifferent operational stages 5110, 5115, 5120, 5125, 5130, 5135, 5140,and 5145 of this UI 5105. FIG. 54 illustrates a UI 5405 of a remotedevice 5400 that receives the switch camera request from the localdevice 5100. FIG. 54 illustrates six different operational stages 5410,5415, 5420, 5425, 5430, and 5435 of the UI 5405.

As shown in FIG. 50 , the process 5000 begins by starting (at 5005) avideo conference between the local and remote devices. The process 5000then (at 5010) receives images from one camera of each device (e.g.,from the front camera of each device) and generates a composite view forthe video conference based on these images. At 5010, the process 5000also sends a video image from the local device to the remote device.

Next, the process 5000 determines (at 5015) whether a request to end thevideo conference has been received. As described above, a videoconference can end in some embodiments at the request of a user of thelocal or remote device. When the process 5000 receives a request to endthe video conference, the process 5000 ends.

When the process 5000 does not receive a request to end the videoconference, the process 5000 then determines (at 5020) whether the userof the device on which the process 5000 is executing (i.e., the user ofthe local device) has directed the device to request that the remotedevice switch between its cameras for the video conference. The process5000 returns to operation 5010 when the process 5000 determines (at5020) that it has not been directed to initiate a remote switch camera.When the process 5000 determines (at 5020) that it has been so directed,the process 5000 transitions to 5025, which will be described furtherbelow.

The first four stages 5110, 5115, 5120, and 5125 of the UI 5105 of FIG.51 illustrate an example of receiving a user's request to switch camerasof the remote device. The first and second stages 5110 and 5115 are thesame as the first and second stages 4610 and 4615 of FIG. 46 . The thirdstage 5120 is the same as the third stage 4620 except the third stage5120 includes a selectable UI item 5180 for a request to the remotedevice 5100 to switch cameras in addition to the selectable UI item 5175for requesting the local device 5100 to switch cameras. The fourth stage5125 illustrates the user of the local device 5100 selecting the UI item5180 (e.g., through a single finger tap 5170 of the selectable UI item5180) for requesting the remote device to switch cameras. The selectionis indicated by the highlighting of the selectable UI item 5180. FIG. 51shows one example of performing this operation, but other embodimentsmay differently perform the operation for requesting the remote deviceto switch cameras.

The example described above by reference to FIG. 51 invokes a remoteswitch camera operation through a remote switch camera user interface.Other embodiments invoke a remote switch camera operation differently.For instance, some embodiments invoke the switch camera operation byhaving a switch camera selectable UI item permanently displayed on a UIduring a video conference such as the UI 5105 of FIG. 52 . In FIG. 52 ,a remote switch camera button 5288 is shown in a display area 1155 alongwith a mute button 5282, an end conference button 5284, and a localswitch camera button 5286.

FIG. 52 illustrates the remote switch camera operation of the UI 5105 ofthe device 5100 in terms of six different stages 5110, 5290, 5130, 5135,5140, and 5145. The first stage 5110 of FIG. 52 is similar to the firststage 5110 of FIG. 51 except that the layout of the display area 1155shows a mute button 5282, a local switch camera button 5286, a remoteswitch camera button 5288, and an end conference button 5284. The secondstage 5290 illustrates the UI 1105 after the user of the local device5100 selects (e.g., through a single finger tap 5170) the remote switchcamera selectable UI item 5288. The last four stages of FIG. 52 aresimilar to the last four stages of FIG. 51 except the layout of thedisplay area 1155 is the same as the layout described above in the firststage 5110 and therefore will not be further described in order to notobscure the description of the invention with unnecessary detail.

Some embodiments provide a similar layout as the one illustrated in FIG.52 except the remote switch camera selectable UI item is displayed inPIP display 5165 instead of the display area 1155. FIG. 53 illustratessuch a layout 5105. In particular, the figure shows the PIP display withthe remote switch camera selectable UI item 5180 and the display area1155 with only a mute button 5282, a local switch camera button 5286,and an end conference button 5284.

As mentioned above, the process 5000 transitions to 5025 when the userrequests a remote switch camera. At 5025, the process 5000 sends therequest to switch cameras to the remote device. In some embodiments,this request is sent through the video conference control channel thatis multiplexed with the audio and video channels by the VTP Manager 3125as described above.

After the request to switch cameras is received, the process 5000determines (at 5030) whether the remote device has responded to therequest to switch cameras. In some embodiments, the remote deviceautomatically sends an accept response (i.e., sends an acknowledgement)to the local device through the video-conference control channel. Inother embodiments, however, the user of the remote device has to acceptthis request through the user interface of the remote device.

The first two stages 5410 and 5415 of the UI 5405 of FIG. 54 illustratean example of the remote user accepting a request to switch cameras ofthe remote device 5400. The first stage 5410 shows (1) a display area5440 for displaying text that notifies the remote user of the request,(2) a selectable UI item 5465 (e.g., allow button 5465) for acceptingthe request to switch cameras of the remote device, and (3) a selectableUI item 5470 (e.g., reject button 5470) for rejecting the request toswitch cameras of the remote device. The second stage 5415 thenillustrates the UI 5405 after the user of the remote device has selected(e.g., through a single finger tap 5480) the UI item 5465 for acceptingthe request to switch cameras, as indicated by the highlighting of theselectable UI item 5465.

When the process 5000 determines (at 5030) that it has not yet receiveda response from the remote device, the process 5000 determines (at 5035)whether a request to end the video conference has been received. If so,the process 5000 ends. Otherwise, the process receives (at 5040) imagesfrom the currently used cameras of the remote and local devices,generates a composite view for the video conference based on theseimages, transmit the local device's video image to the remote device,and then transitions back to 5030.

When the process 5000 determines (at 5030) that it has received aresponse from the remote device, it determines (at 5045) whether theremote device accepted the request to switch cameras. If not, theprocess 5000 returns to operation 5010 to continue to receive imagesfrom the camera of the other device. Otherwise, the process receives (at5050) images from the other camera of the remote device and thenperforms (at 5055) a switch camera animation on the local device todisplay a transition between the video of the previously utilized remotecamera and the video of the currently utilized remote camera (i.e., thereceived images at operation 5050). After 5055, the process transitionsback to 5010, which was described above.

The last four operational stages 5130, 5135, 5140, and 5145 that areillustrated for the UI 5105 in FIG. 51 illustrate one example of such aremote switch camera animation on the local device 5100. The exampleanimation is similar to the example animation illustrated in the stages4815, 4820, 4825, and 4830 of FIG. 48 except FIG. 51 shows in thedisplay area 5150 an animation that replaces the video of a woman thatis captured by the front camera of the remote device with the video of atree that is captured by the back camera of the remote device. The lastfour stages of FIG. 52 and FIG. 53 illustrate the same animation as theone in FIG. 51 except the display area 1155 of FIGS. 52 and 53 containsdifferent selectable UI items than the display area 1155 in FIG. 51 .

In some embodiments, when the remote device switches cameras, the UI ofthe remote device also performs a switch camera animation to display atransition between the two cameras. The last four operational stages5420, 5425, 5430, and 5435 that are illustrated for the UI 5405 in FIG.54 illustrate an example of a switch camera animation that is displayedon the remote device 5400 when the remote device 5400 switches betweencameras. This animation is similar to the animation illustrated in thestages 4630, 4635, 4640, and 4645 of FIG. 46 except that the animationin the display area 5445 replaces the video of a woman that is capturedby the front camera of the remote device 5400 with the video of a treethat is captured by the back camera of the remote device 5400.

As noted above, FIGS. 46, 47, 48, 49, 51, 52, 53, and 54 show variousexamples of switch camera animations performed on a user interface. Insome embodiments, the switch camera animation causes changes to theimage processing operations of the respective dual camera mobile devicesuch as scaling, compositing, and perspective distortion, which can beperformed by the video conference manager 1604 and the image processingmanager 1608, for example.

C. Exposure Adjustment

During a video conference between a dual camera mobile device andanother mobile device, different embodiments provide differenttechniques for adjusting the exposure of images captured by cameras ofeither mobile device. Some embodiments provide techniques for a user ofthe dual camera mobile device to adjust the exposure of images capturedby a camera of the other device while other embodiments providetechniques for the user to adjust the exposure of images captured by acamera of the dual camera mobile device. Several example techniques willbe described in detail below.

FIG. 55 illustrates a process 5500 for performing a remote exposureadjustment operation on a dual camera mobile device of some embodimentsduring a video conference. In the following discussion, the devicethrough which a user directs a remote device to adjust its exposurelevel is referred to as the local device. In some embodiments, theprocess 5500 is performed by the video conference manager of the localdevice. In addition, the process 5500 will be described by reference toFIGS. 56, 57, and 58 , which illustrate various ways for the user of thelocal device to request the remote device to perform an exposureadjustment operation.

As shown in FIG. 55 , the process 5500 begins by starting (at 5505) avideo conference between the local and remote devices. The process 5500then receives (at 5510) a video from the remote device for display onthe display screen of the local device. Next, the process 5500determines (at 5515) whether a request to end the video conference hasbeen received. As described above, some embodiments can receive arequest to end the video conference from a user of the local or remotedevice. When the process 5500 receives a request to end the videoconference, the process 5500 ends.

However, when the process 5500 does not receive a request to end thevideo conference, the process 5500 then determines (at 5520) whether arequest for adjusting the exposure of the remote device's camera hasbeen received. When the process 5500 determines that a request foradjusting the exposure of the remote device's camera has not beenreceived, the process 5500 returns back to operation 5510 to receiveadditional video captured from the remote device. FIGS. 56, 57, and 58illustrate three different examples of providing a way for a user tomake such a request. In FIGS. 56, 57, and 58 , the first stages 5610,5710, and 5810 all show PIP displays 5625, 5750, and 5835 of the localdevices 5600, 5700, and 5800 that display two videos: one captured by acamera of the local device and the other captured by a camera of theremote device. In first stages 5610, 5710, and 5810 the man in thebackground display 5635, 5760, and 5845 is dark, indicating that the manis not properly exposed.

The second stage 5615 of FIG. 56 illustrates one way for the user of thelocal device 5600 to request the remote device to perform an exposureadjustment by selecting the remote device's video (e.g., through asingle tap on the background display 5635). In this way, the UI 5605automatically associates the user's selection of a region of interestdefined by a box 5645 with the user's desire to direct the remote deviceto perform an exposure adjustment on the region of interest and thusdirects the video conference manager of the local device to contact theremote device to perform an exposure adjustment operation. The definedregion of interest is used by the remote device in the calculation ofthe exposure adjustment.

Like the second stage 5615 of FIG. 56 , the second stage 5715 of FIG. 57shows the local user's selection of the remote device's video exceptthis selection directs the UI 5705 to display a selectable UI item 5770as shown in the third stage 5720. The fourth stage 5725 illustrates theuser of the local device selecting the selectable UI item 5770 to directthe remote device to perform an exposure adjustment operation asdescribed above.

The second stage 5815 of FIG. 58 is similar to the second stage 5715 ofFIG. 57 , but instead of the user's selection of the remote device'svideo directing the UI to display a single selectable UI item, theuser's selection directs the UI 5805 to display a menu of selectable UIitems 5855, 5860, 5865, and 5870, as shown in the third stage 5820. Theselectable UI items include an Auto Focus item 5855, an Auto Exposureitem 5860, a Switch Camera item 5865, and a Cancel item 5870. In someembodiments, the Switch Camera selectable UI item 5865 is used torequest a local switch camera operation while in other embodiments theSwitch Camera selectable UI item 5865 is used to request a remote switchcamera operation. The fourth stage 5825 illustrates the user selectingthe Auto Exposure item 5860 to direct the remote device to perform anexposure adjustment operation as described above.

When the process 5500 determines (at 5520) that the local user directedthe local device to request an exposure adjustment operation, theprocess 5500 sends (at 5525) a command to the remote device through thevideo conference control channel to adjust the exposure of the videocaptured by the camera that is currently capturing and transmittingvideo to the local device. After operation 5525, the process 5500transitions back to operation 5510, which is described above.

In some embodiments, the user of the remote device is required toprovide permission before the remote device performs an exposureadjustment operation, while in other embodiments the remote deviceperforms the exposure adjustment operation automatically upon receivingthe request from the local device. Moreover, in some embodiments, someof the video conference functionalities are implemented by the videoconference manager 1604. In some of these embodiments, the videoconference manager 1604 performs the exposure adjustment operation byinstructing the CIPU 1650 to adjust the exposure setting of the sensorof the remote device camera being used.

The last stages 5620, 5730, and 5830 of FIGS. 56, 57, and 58 show theremote device's video lighter, which indicates that the man is properlyexposed. Although FIGS. 56, 57, and 58 provide examples of receiving anexposure adjustment request to correct the exposure of a remote device,some embodiments provide ways for user of the local device to requestthat the local device adjust the exposure of a camera of the localdevice. Such a request can be made similar to the ways illustrated inFIGS. 56, 57, and 58 for requesting a remote device to adjust itscamera's exposure.

FIGS. 56-58 described above show several user interfaces for performingexposure adjustment operations. In some embodiments, the exposureadjustment operation can cause changes to the image processingoperations of the dual camera mobile device such as invoking theexposure adjustment process 5900, which is described in further detailbelow. The exposure adjustment operation can also cause changes to theoperation of the camera of the dual camera mobile device that iscapturing the video like changing the exposure level setting of thecamera, for example.

1. Exposure Adjustment Methodology

FIG. 59 conceptually illustrates an exposure adjustment process 5900performed by an image processing manager of some embodiments such asthat illustrated in FIG. 16 . In some embodiments, the process 5900 ispart of the exposure adjustment operations described above by referenceto FIGS. 55, 56, 57, and 58 . In some of such embodiments, the imageprocessing manager 1608 performs the process 5900 and adjusts a camera'sexposure setting by sending instructions to the video conference manager1604, which instructs the CIPU 1650 to adjust the camera sensor 405 a or405 b, as mentioned above.

In some embodiments, the process 5900 is performed by the imageprocessing layer 930 shown in FIG. 9 while in other embodiments theprocess 5900 is performed by the statistics engine 465 shown in FIG. 4 .Some embodiments perform the process 5900 on images captured by camerasof (local or remote) devices in a video conference while otherembodiments perform the process 5900 as part of the process 2100 (e.g.,operation 2110) illustrated in FIG. 21 . Some embodiments perform anexposure adjustment operation to expose images captured by the camerasof the dual camera mobile device that are not too light and not toodark. In other words, the process 5900 is performed to capture images ina manner that maximizes the amount of detail as possible.

The process 5900 begins by receiving (at 5905) an image captured by acamera of the dual camera mobile device. In some embodiments, when thereceived image is a first image captured by a camera of a device in avideo conference, the process 5900 is not performed on the first image(i.e., there was no image before the first image from which to determinean exposure value). The process 5900 then reads (at 5910) pixel valuesof a defined region in the received image. Different embodiments defineregions differently. Some of such embodiments define differently shapedregions such as a square, a rectangle, a triangle, a circle, etc. whileother of such embodiments define regions in different locations in theimage such as center, upper center, lower center, etc.

Next, the process 5900 calculates (at 5915) an average of the pixelvalues in the defined region of the image. The process 5900 determines(at 5920) whether the calculated average of the pixel values is equal toa particular defined value. Different embodiments define differentparticular values. For example, some embodiments define the particularvalue as the median pixel value of the image's dynamic range. In someembodiments, a range of values is defined instead of a single value. Insuch embodiments, the process 5900 determines (at 5920) whether thecalculated average of the pixel values is within the define range ofvalues.

When the calculated average of the pixel values is not equal to theparticular defined value, the process 5900 adjusts (at 5925) theexposure value based on the calculated average. When the calculatedaverage of the pixel values is equal to the particular defined value,the process 5900 ends. In some embodiments, an exposure value representsan amount of time that a camera sensor is exposed to light. In someembodiments, the adjusted exposure value is used to expose the nextimage to be captured by the camera that captured the received image.After the exposure value is adjusted based on the calculated average,the process 5900 ends.

In some embodiments, the process 5900 is repeatedly performed until thecalculated average of pixel values is equal to the particular definedvalue (or falls within the defined range of values). Some embodimentsconstantly perform the process 5900 during a video conference whileother embodiments perform the process 5900 at defined intervals (e.g., 5seconds, 10 seconds, 30 seconds, etc.) during the video conference.Furthermore, during the video conference, the process 5900 of someembodiments dynamically re-defines the particular pixel value beforeperforming the process 5900.

FIG. 60 conceptually illustrates examples of exposure adjustmentoperations of some embodiments. Each of the examples 6000, 6010, and6015 shows an image 6020 captured by a camera of the dual camera mobiledevice on the left side. Specifically, the image 6020 shows a darkperson in front of a sun. The dark person indicates that the exposurelevel of the image is not high enough to expose the person's face orbody. The right side of each example 6000, 6010, and 6015 shows an image6025, 6030, and 6035, respectively, captured after the image 6020. Insome embodiments, the image 6020 and the images on the right side areimages of a video captured by the camera of the dual camera mobiledevice. In other embodiments, the image 6020 and the image on the rightside are still images captured by the camera of the dual camera mobiledevice at different instances in time.

The first example 6000 illustrates an operation with no exposureadjustment. As such, the image 6025 appears the same as the image 6020.Since no exposure adjustment was performed, the person in the image 6025remains dark like the person in the image 6020.

In the second example 6010, an exposure adjustment operation isperformed on the image 6020. In some embodiments, the exposureadjustment operation is performed by the process 5900 using the definedregion 6040. Based on the exposure adjustment operation, the exposurelevel of the camera is adjusted and the camera captures the image 6030using the adjusted exposure level. As shown in FIG. 60 , the person inthe image 6030 is not as dark as the in the image 6025. However, theperson's face and body in the image 6030 is still not clear.

The third example 6015 shows an exposure adjustment operation performedon the image 6020. Similar to the second example 6010, the exposureadjustment operation of the example 6015 of some embodiments isperformed by the process 5900 using the defined region 6045. Based onthe exposure adjustment operation, the exposure level of the camera isadjusted and the camera captures the image 6035 using the adjustedexposure level. As seen in FIG. 60 , the person in the image 6035 isperfectly exposed since the person's face and body is visible.

In some embodiments, the selection of the defined region may be made bythe user of the dual camera mobile device. The device itself may alsoautomatically adjust its defined region for the exposure adjustmentoperation through the feedback loop for exposure adjustment mentionedabove in the CIPU 400. The statistics engine 465 in FIG. 4 may collectdata to determine whether the exposure level is appropriate for theimages captured and adjust the camera sensors (e.g., though a directconnection to the sensor module 415) accordingly.

D. Focus Adjustment

FIG. 61 illustrates a process 6100 for adjusting the focus of a dualcamera mobile device during a video conference. In the followingdiscussion, the device through which a user directs a remote device toadjust its camera focus is referred to as the local device. The process6100 of FIG. 61 is in some embodiments performed by the video conferencemanager 1604 of the local device. Also, this process will be describedbelow by reference to FIGS. 62 and 63 , which provide two exemplarymanners for the user of the local device to request a focus adjustmentoperation to be performed by the remote device.

As shown in FIG. 61 , the process 6100 begins by starting (at 6105) avideo conference between the local and remote devices. The process 6100then receives (at 6110) a video from the remote device for display onthe display screen of the local device. Next, at 6115, the process 6100determines whether a request to end the video conference has beenreceived. As described above, a video conference can end in someembodiments at the request of a user of the local or remote device. Whenthe process 6100 receives a request to end the video conference, theprocess 6100 ends.

Otherwise, the process determines (at 6120) whether it has received arequest for adjusting the focus of the remote camera of the remotedevice. When the process 6100 determines that it has not received arequest for adjusting the focus of the remote camera of the remotedevice, the process 6100 returns to operation 6110 to receive additionalvideo from the remote device. FIGS. 62, 63, and 64 illustrate threedifferent ways that different embodiments provide to a user to make sucha request. In FIGS. 62, 63, and 64 , the first stages 6210, 6310, and6472 all show a PIP display 6225, 6335, and 6482 of the local device6200, 6300, and 6471 that displays two videos, one captured by the localdevice, and the other captured by the remote device. The display areas1155 and 1155 in FIGS. 62 and 63 show an end conference button. However,in FIG. 64 , the layout of the display area 1155 is the same as thelayout of the display area 1155 of FIG. 12 , described above. Moreover,the switch camera button 6488 shown in the display area 1155 can beselected to invoke a local switch camera operation in some embodimentsor a remote switch camera operation in other embodiments. As shown inthe first stages 6210, 6310, and 6472, the video of the remote devicethat is displayed in the background display 6235, 6345, and 6480 isblurry.

The second stage 6215 of FIG. 62 illustrates an approach whereby theuser of the local device requests a focus adjustment from the remotedevice by simply selecting the remote device's video (e.g., through asingle tap 6240 on the remote device's video). Under this approach, theUI 6205 automatically associates the user's selection of a region ofinterest defined by a box 6245 with the user's desire to direct theremote device to perform an operation (such as focus) on the region ofinterest and therefore directs the video conference manager 1604 of thelocal device 6200 to contact the remote device to perform an adjustmentoperation (such as an focus adjustment operation). The defined region ofinterest is used by the remote device in the calculation of the focusadjustment.

The second stage 6315 of FIG. 63 similarly shows the local user'sselection of the remote video (e.g., through the user's tapping of theremote device's video). However, unlike the example illustrated in FIG.62 , this selection in FIG. 63 directs the UI 6305 to display a menu ofselectable UI items 6355, 6360, 6365 and 6370 (which can be implementedas selectable buttons), as shown in the third stage 6320. Theseselectable UI items include an Auto Focus item 6360, an Auto Exposureitem 6365, a Switch Camera item 6370 and a Cancel item 6355. In someembodiments, the Switch Camera selectable UI item 6370 is used torequest a local switch camera operation while in other embodiments theSwitch Camera selectable UI item 6370 is used to request a remote switchcamera operation. The fourth stage 6325 then illustrates the local userselecting the auto-focus item 6360.

The second stage 6474 of FIG. 64 again similarly shows the local user'sselection of the remote video (e.g., through the user's tapping of theremote device's video). However, unlike the example illustrated in FIG.63 , this selection in FIG. 64 directs the UI 6478 to request a focusadjustment operation (i.e., in second stage 6474). After the focusadjustment operation is completed, the UI 6478 displays a menu ofselectable UI items 6484 and 6486 (i.e., in third stage 6476), which canbe implemented as selectable buttons. These selectable UI items includean Auto Exposure item 6486 and a Cancel item 6484.

When the process determines (at 6120) that the local user directed thelocal device to request a focus adjustment operation, the process 6100sends (at 6140) a command to the remote device through the videoconference control channel to adjust the focus of the camera whose videothe remote device is currently capturing and transmitting. After 6140,the process transitions back to 6110, which was described above.

In some embodiments, the user of the remote device has to providepermission before the remote device performs this operation, while inother embodiments the remote device performs this operationautomatically upon receiving the request for the local device. Also, insome embodiments, the focus adjustment operation adjusts the focussettings of the remote device's camera that is being used during thevideo conference. In some of such embodiments, some of the videoconference functionalities are implemented by the video conferencemodule 1602 as discussed above. In these embodiments, the videoconference manager 1604 instructs the CIPU 1650 to adjust the sensor ofthe remote device camera being used.

The last stages 6220, 6330, and 6476 of FIGS. 62, 63, and 64 show theremote device's video properly focused. Although FIGS. 62, 63, and 64provide examples of receiving a focus adjustment request to correct thefocus of a remote device, some embodiments allow the local device's userto request that the local device adjust the focus of a camera of thelocal device. Such a request can be made similar to the approaches shownin FIGS. 62, 63, and 64 to requesting a remote device to adjust itscamera's focus.

FIGS. 62, 63, and 64 illustrate three example user interfaces that allowa user to perform a focus adjustment operation. In some embodiments, thefocus adjustment operation causes changes to the operation of the cameraof the dual camera mobile device that is capturing the video displayedin the UIs such as changing the focus of the camera.

As discussed above in FIGS. 56 and 62 , the defined region of interestwas used by the remote mobile device in the computation for exposureadjustment and focus adjustment of the videos, respectively. However, insome other embodiments, the user's selection of a region of interest maybe used to direct the remote device to perform one or more operations.For example, in some embodiments, both exposure adjustment and focusadjustment may be performed based on the defined region of interest,thereby directing the remote device to perform both operations.

E. Frame Rate Control

During a video conference, some embodiments may wish to adjust ormaintain the rate at which images of a video captured by a camera of thedual camera mobile device are transmitted (i.e., frame rate) to theother device in the video conference. For example, assuming a fixedbandwidth, some of such embodiments reduce the frame rate of the videoto increase the picture quality of the images of the video while otherof such embodiments increase the frame rate of the video to smooth outthe video (i.e., reduce jitter).

Different embodiments provide different techniques for controlling theframe rate of images of a video during the video conference. One examplepreviously described above adjusts the VBI of the sensor module 415 fora camera in order to control the rate at which images captured by thecamera are processed. As another example, some embodiments of themanagement layer 935 of the video conference module 925 shown in FIG. 9control the frame rate by dropping images. Similarly, some embodimentsof the image processing layer 930 control the frame rate by droppingimages. Some embodiments provide yet other techniques for controllingframe rates such as dropping frames in the universal transmission buffer3120.

V. Dual Camera

A. Combined Views

1. Picture-In-Picture: Display of Two Remote Cameras

Some embodiments allow a dual camera mobile device to display videoscaptured from the mobile device and videos captured from another dualcamera mobile device during a video conference in any of several displayarrangements. FIG. 65 illustrates examples of different displayarrangements for videos captured from one or more dual camera mobiledevices. In FIG. 65 , a user of a dual camera mobile device 6500 (DeviceA) and a second user of a second dual camera mobile device 6505 (DeviceB) are having a video conference with each other.

FIG. 65 shows four examples of display arrangements for Device A on theleft. The four display arrangements for Device A are the First View6510, the Second View 6515, the Third View 6520, and the Fourth View6525. In addition, FIG. 65 also shows four examples of displayarrangements for Device B on the right. The four display arrangementsfor Device B are the First View 6565, the Second View 6570, the ThirdView 6575, and the Fourth View 6580. In this example, Device A onlydisplays the two videos captured from the cameras of Device A whileDevice B displays the two videos captured from the cameras of Device Aas well as one or both of the videos captured from the cameras of DeviceB.

In the first view 6510, a UI 6585 of Device A provides a compositedisplay 6512. The composite display 6512 includes two display areas: adisplay area 6530 for displaying video captured from Device A's backcamera and a display area 6535 for displaying video captured from DeviceA's front camera. In this example, The display area 6530 is located inthe upper half of the composite display 6512 while the display area 6535is located in the lower half of the composite display 6512. The twodisplay areas are of equal size in the first view 6510. The upperdisplay area 6530 is displaying a video of a mountain, which is assumedto be a mountain that is being captured by Device A's back camera. Thedisplay area 6535 is displaying a tree and a man with a hat, which areassumed to be a tree and a man that are being captured by Device A'sfront camera.

The UI 6585 in the second view 6515 provides a composite display 6517that includes the same two display areas from the first view 6510,except that the display area 6535 (displaying video captured from DeviceA's front camera) is now located in the upper half of the compositedisplay 6517 and the display area 6530 (displaying video captured fromDevice A's back camera) is located in the lower half of the compositedisplay 6517.

In the third view 6520, the UI 6585 provides a PIP display 6595. The PIPdisplay 6595 includes two display areas: the display area 6535displaying video captured from Device A's front camera as a backgrounddisplay area and the display area 6530 displaying video captured fromDevice A's back camera as a foreground inset display area. In this view,the background display area 6535 takes up a majority of the PIP display6595 while the inset display area 6530 is smaller and overlaps a portionof the background display area 6535.

The UI 6585 in the fourth view 6525 also presents a PIP display 6598that includes the display areas 6530 and 6535 as shown in the third view6520. Unlike the PIP display 6595, the PIP display 6598 includes thedisplay area 6530 (captured from Device A's back camera) as thebackground main display and the display area 6535 (captured from DeviceA's front camera) as the foreground inset display. In addition, the PIPdisplay 6598 is presented in landscape view (i.e., the width of the PIPdisplay 6598 is longer than the height).

The above examples illustrate four possible composite views for theDevice A's UI—two in which the two display areas 6530 and 6535 fordisplaying the two cameras of the first device are tiered vertically andtwo PIP views. Other views are also possible for Device A's UI. Forexample, the two display areas could be tiered horizontally ordiagonally, or different PIP views could be used.

The various views illustrated for Device B show that different views forthe UI of device B are possible. These views include video captured fromboth cameras of Device A as well as one or more cameras of Device B. Inthe first view 6565 of Device B, a UI 6590 of Device B provides a PIPdisplay 6568. The PIP display 6568 includes a composite display area6569 that is identical to the composite display 6512 displayed on DeviceA, as well as an inset display area 6550 that displays video captured byone of Device B's cameras (e.g., the front camera). The compositedisplay area 6569 includes a display area 6531 for displaying videocaptured from Device A's back camera and a display area 6536 fordisplaying video captured from Device B's front camera. The compositedisplay 6569 displaying video from Device A takes up the majority of thePIP display 6568 while the inset display area 6550 is smaller andoverlaps the composite display 6569. The display area 6550 is displayinga video of a smiley face, which is assumed to be a smiley face whosevideo is being captured by Device B's front camera.

The UI 6590 of Device B in the second view 6570 provides a PIP display6572. The PIP display 6572 includes the display area 6550 (displayingvideo captured from Device B's front camera) and a composite display6573 with the display areas 6531 and 6536 displaying video captured fromthe cameras of Device A. The composite display 6573 is identical to thecomposite display 6517 in the second view 6515 for Device A and takes upa majority of the PIP display 6572. Like in the PIP display 6568 in thefirst view 6565, the display area 6550 is smaller and overlaps thecomposite display 6573. Specifically, in both views the display areaoverlaps a portion of the display area 6531 that displays video capturedfrom Device A's back camera.

In the third view 6575, the UI 6590 provides a PIP display 6577 that issimilar to the PIP display 6595 in the third view 6520 for Device A. ThePIP display 6577 also includes the additional display area 6550 as asecond inset display area that overlaps the background display area6536. The two inset display areas 6531 and 6550 are tiled horizontallyat the bottom of the background primary display area 6536.

The UI 6590 in the fourth view 6580 provides a composite display 6582.The composite display 6582 includes three displays: a PIP display 6583,the display area 6550, and a display area 6540 (e.g., for displayingvideo captured by Device B's back camera). The PIP display 6583 isidentical to the PIP display 6598 in the fourth view 6525 for Device Aand takes up a majority of the composite display area 6582. The displays6540 and 6550 are smaller and tiled horizontally below the PIP displayarea 6583.

While FIG. 65 illustrates four possible views for Device B, many otherviews are possible. The background composite display of video fromDevice A could be tiled horizontally rather than vertically, the insetcould overlap the front camera display area of Device A rather than theback camera display area, the larger display areas could be displayingthe Device B camera(s) rather than those of Device A, the insets couldbe located differently, etc.

Each set of arrows 6560 stemming from each view of Device A demonstratesthat there is no requirement of a correlation between the display shownon Device A and the display shown on Device B. For instance, even ifDevice A is displaying its video in the arrangement of view 6510 (e.g.,according to a selection of that arrangement by the user of Device A),Device B could be displaying video in any of the four illustratedarrangements or in any of a number of other arrangements not shown inFIG. 65 (e.g., according to a selection of that arrangement by the userof Device B). Put another way, the display arrangement for Device A isindependent of the display arrangement of Device B. Some embodiments donot transmit display areas from one device to another but rather justtransmit the video (e.g., in encoded form), which is displayed in itscorresponding display area by the device.

2. Specialized PIPs

Some embodiments allow a user of a dual camera mobile device tosuperimpose a foreground of a video onto another video in a PIP displayduring a video conference. In some embodiments, the foreground of avideo blends into the other video in such a way that they appear as adisplay of a single video captured by a single camera. FIG. 66illustrates an example of such superimposing of a foreground of an insetvideo onto a background video in a PIP display.

FIG. 66 illustrates this video superimposition operation in terms ofseven operational stages 6620, 6625, 6630, 6635, 6640, 6660, and 6665 ofa UI 6670. The first stage 6620 illustrates the UI 6670 of a dual cameramobile device 6600 with a PIP display 6682 during a video conferencewith a remote device. As shown in the first stage 6620, the PIP display6682 includes two video displays: a background main display 6610 and aforeground inset display 6605. The background main display 6610 takes upa majority of the UI 6670, while the foreground inset display 6605 issmaller and overlaps the background main display 6610.

In this example, the background display area 6610 is displaying a videoof a mountain, which is assumed to be a mountain that is being capturedby one of the remote device's cameras. The foreground inset display area6605 is displaying a video of a person with a hat, which in this exampleis assumed to be a person whose video is being captured by one of thelocal device's cameras. Below the PIP display 6682 is a selectable UIitem 6685 labeled “End Conference” (e.g. a button 6685) that allows theuser to end the video conference with a selection of the item (e.g., bysingle- or double-tapping the button).

The second stage 6625 illustrates the invocation of a selectable menu6675. In some embodiments, the menu of selectable UI items 6675 may beinvoked by selecting (e.g., by touching) the PIP display area 6682.Instead of, or in conjunction with, such an invocation operation, someembodiments also allow the user to invoke the menu of selectable UIitems 6675 through other operations, such as through differenttouchscreen operations or using one or more other physical inputs of thedevice.

The third stage 6630 displays the UI 6670 with the invoked set ofselectable UI items for selecting the video superimposition operation.In this example, a pop-up menu 6675 with several selectable UI items isdisplayed over the PIP display 6682. The menu of selectable UI items6675 includes a “Flip PIP” selectable UI item 6640 (e.g. button 6640), a“Specialized PIP” selectable UI item 6645 (e.g. button 6645), and a“Cancel” selectable UI item 6690 (e.g. button 6690). In this example,selecting the “Flip PIP” button 6640 would cause the UI 6670 to swap thebackground display 6610 with the inset display 6605 (as will bediscussed in detail in the next section), selecting the “SpecializedPIP” button 6645 would cause the UI 6670 to begin the operation of videosuperimposition, and selecting the “Cancel” button 6690 would remove thepop-up menu 6675 from the PIP display 6682. Other embodiments includedifferent or more items in the PIP pop-up menu 6675.

The fourth stage 6635 illustrates the UI 6670 after the user hasselected the “Specialized PIP” button 6645 (e.g., by tapping on thebutton 6645 with his finger 6695). This selection is indicated by thehighlighting of the button 6645 on the UI display 6670. Some embodimentsuse different indication displays (e.g., highlighting the border of theselected item or the text in the selected item).

The fifth stage 6640 shows the UI 6670 after the video superimpositionoperation has begun. In this stage, the UI 6670 allows the user tochoose from which video he wants to extract as a foreground and whichvideo he wants to use as a background in the superimposed video. The UI6670 provides the options through a pop-up menu 6680 with severalselectable UI items displayed over the PIP display 6682. The pop-up menu6680 of selectable UI items includes a “Select Inset” selectable UI item6655 (e.g. button 6655), a “Select Main” selectable UI item 6650 (e.g.button 6650), and a “Cancel” selectable UI item 6692 (e.g. button 6692).

Selection of the “Select Inset” button 6655 would cause the UI 6670 tosuperimpose the foreground of the inset video 6605 from the localdevice's camera (i.e., the man with a hat) onto the background mainvideo 6610 from the remote device's camera. On the other hand, selectionof the “Select Main” button 6650 would cause the UI 6670 superimpose theforeground of the background main video 6610 from the remote device'scamera (i.e., the mountain) onto the inset video 6605 from the localdevice's camera. In some embodiments, this causes a switch of the twovideo feeds such that the video currently in the inset display area 6605will occupy most of the UI 6670 and the video currently in the primarydisplay area 6610 will be superimposed on the now-primary video.Selection of the “Cancel” button 6692 would abort the videosuperimposition operation and remove the pop-up menu 6680 from the PIPdisplay area 6682.

The sixth stage 6660 illustrates the UI 6670 after the user has selectedthe “Select Inset” button 6655 (e.g., by tapping on the button 6655 withhis finger 6695). This selection is indicated by the highlighting of thebutton 6655 on the UI display 6670. Some embodiments use differentindication displays (e.g., highlighting the border of the selected itemor the text in the selected item).

The seventh stage 6665 illustrates the UI 6670 after the videosuperimposition operation is complete. As shown in the UI 6670, theforeground of the inset display area 6605 (i.e., the man with a hat) isextracted from the display area 6605. The window frame and thebackground (i.e., everything else other than the foreground) of theinset display 6605 are also removed from the screen. Finally, theforeground (i.e., the man with a hat) is blended into the backgroundvideo 6610 in such a way that it appears as a single video. Variousdifferent techniques may be used to remove the background of the insetvideo. Some embodiments identify pixels that are not moving relative toother pixels, look for patterns or colors that are constant, use abaseline image compared to the image that includes the foreground andsubtract out the difference, or use a different technique.

While the example of FIG. 66 illustrates the foreground of the insetdisplay area 6605 staying in the same place in the UI 6670 whensuperimposed onto the background display area 6610, this is only oneexample of how the superimposition can work. Some embodiments move theforeground video to a particular location in the UI 6670 (e.g., thecenter, one of the corners, etc.). Similar to the features shown inSections IV.A.1 and IV.A.3, some embodiments allow the user of the localdevice to drag the superimposed foreground video around in the UI orchange the size of the superimposed foreground video.

Different techniques may be used to determine which portion(s) of videoimages is the “foreground” for the video superimposition operationdescribed above. One such method of some embodiments determines whichportion(s), if any, of the video images is dynamic. The dynamic portionis considered the “foreground” because the background of video images isgenerally static (i.e., no motion). In such embodiments, video imagesare analyzed over a particular period of time. If the difference among aparticular pixel's values over the particular period is not greater thana defined threshold value (e.g., 5%, 10%, 15%), the particular pixel isconsidered a static pixel. After each pixel in the video images isanalyzed, the dynamic pixels (i.e., not static) of the video images areconsidered the “foreground” of the video images.

FIG. 67 illustrates an example of such technique for determining theforeground of video images that can be performed by the video conferencemanager 1604 or the image processing manager 1608, for example.Specifically, FIG. 67 illustrates a sequence of six images 6705-6730 ofa video that shows a person with a hat and a tree. In this example, itis assumed that the person is not standing entirely still and may betalking. As described above, each pixel in the video images is analyzedto determine whether the pixel is dynamic or static. For instance, thedifference among pixel 6735's value in images 6705-6730 is determinedwhether it is greater than a defined threshold. Here, since the pixel6735 represents part of the ground rather than the person, the pixel6735 is considered static. After all of the pixels in the images6705-6730 are analyzed, it is determined that the person in the imagesis dynamic and the remaining portion of the images is static. As such,the person is the “foreground” that will be extracted by the operationdescribed by reference to FIG. 66 , above.

3. Swap Videos in a Picture-in-Picture Display

Some embodiments allow the user of a dual camera mobile device to swapthe two display areas in a PIP display (i.e., the inset display areabecomes the background display area, and the background display areabecomes the inset display area in the PIP display) during a videoconference. FIG. 68 illustrates an example of swapping an inset displayarea 6605 with a background display area 6610 in a PIP display 6682during a video conference.

FIG. 68 illustrates the swap PIP operation in terms of eight operationalstages of a UI 6670 of the device 6800 in FIG. 66 . The first threestages in FIG. 68 are identical to the first three stages in FIG. 66 .In these stages, the user has brought up the menu 6675 within the UI6670 through a selection using the touchscreen of the local device.

The fourth stage 6840 in FIG. 68 illustrates the UI 6670 after the userhas selected the “Flip PIP” button 6640 (e.g., by tapping on the button6640 with his finger 6695). This selection is indicated by thehighlighting of the button 6640 on the UI display 6670. Some embodimentsuse different indication displays (e.g., highlighting the border of theselected item or the text in the selected item).

The fifth stage 6845 illustrates the UI 6670 after the swap PIPoperation has started. Some embodiments animate the swapping of theinset and background display 6605 and 6610 through a flipping motion.FIG. 68 illustrates an example of one such animation. In this example,the animation can be described through the flipping of a viewing pane ofwhich the PIP display 6682 (before the swap operation is performed) ison one side and the new PIP display 6684 (after the swap operation isperformed) is on the other side. The viewing pane rotates 180 degreesaround a vertical axis 6686 located in the center of the PIP display6682. At this fifth stage 6845, the viewing pane begins to rotate aboutthe vertical axis 6686.

In the sixth stage 6850, the viewing pane is shown to have rotatedapproximately 90 degrees. This is indicated by the thin line 6688 (i.e.the edge of the viewing pane) displayed in the center of the screen. Theseventh stage 6855 illustrates the rotation of the viewing pane close tocompletion. A new PIP display 6684 starts to appear from the other sideof the viewing pane and expands horizontally to fill the device'sscreen. The PIP display 6684 includes the two display areas 6605 and6610 after the swap operation is performed. The display area 6605presenting the video of a man with a hat (from the local device'scamera) is now in the background of the PIP display 6684 and the display6610 presenting the video of a mountain (from the remote device'scamera) in now the foreground of the PIP display 6684 overlapping thedisplay 6605. The eighth stage 6860 shows the completion of the swapdisplays animation.

One of ordinary skill will recognize that the animation shown in FIG. 68is only one of many possible animations of the PIP inset/background swapoperation. For instance, different embodiments might rotate the viewingpanes along a horizontal axis, instantaneously swap the two displayareas, expand one display area while shrinking the other, etc. Someembodiments provide one animation that is always used for the swapoperations, while other embodiments allow a user to choose from severalanimations or use different animations (e.g., through random selection).Furthermore, the swap operation can cause changes to the imageprocessing operations of the dual camera mobile device such as causingthe video conference manager 1604 to change the scaling and compositingof the videos in response to the user's input.

4. Snap-to-Corner

Some embodiments of the invention allow a user of a dual camera mobiledevice to modify the composite display by moving around one or moredisplay areas that form the composite display. One example of suchmovement is described above in Section IV.A.1. Such movement of insetdisplays is also possible when a PIP display includes more than oneinset display area.

FIG. 69 illustrates such an example that is performed during a videoconference. This example illustrated in FIG. 69 is similar to theexample illustrated in FIG. 3 , except FIG. 69 illustrates moving aroundan inset display area 6910 of a PIP display 6965 that includes two insetdisplay areas 6905 and 6910 rather than only one such inset displayarea.

In FIG. 69 , a UI 6960 of a mobile device 6900 presents a PIP display6965 during a video conference with a remote user of another device. ThePIP display 6965 in FIG. 69 includes three video displays: a backgroundmain display 6915 and two foreground inset displays 6905 and 6910. Inthis example, the background main display 6915 presents a video of aperson singing and playing a guitar, which is assumed to be videocaptured by the remote device's back camera. The foreground insetdisplay 6905 presents a video of a person holding a racket, which inthis example is assumed to be video captured by the local device's backcamera. The other foreground inset display 6910 presents a video of aperson with a hat, which in this example is assumed to be a person whosevideo is being captured by the local device's front camera. Below thePIP display 6965 is a selectable UI item 6970 labeled “End Conference”(e.g., a button 6970) that allows the user to end the video conferenceby selecting the item.

This PIP display 6965 is only one manner of presenting a composite viewof the videos being captured by the remote and local devices. Someembodiments may provide other composite views. For instance, instead ofhaving a larger background display 6915 for the video from the remotedevice, the larger background display 6915 can be of the video from thelocal device and the smaller foreground inset displays 6905 and 6910 canbe of the videos from the remote device. Also, some embodiments allowthe local and remote videos to appear in the UI 6960 with the insetdisplays 6905 and 6910 on one side and the background display 6915 onanother side or all three side-by-side. In other embodiments, the PIPdisplay 6965 may contain a larger background display 6915 and/or asmaller foreground inset display. The manner of the PIP display 6965 ora default display mode may be specified by the user in some embodiments.

FIG. 69 illustrates the movement of one of the two inset display areasin a UI 6960 of a device 6900, by reference to five differentoperational stages 6920, 6925, 6930, 6935, and 6940. The first stage6920 illustrates the UI 6960 during a video conference between the localuser of the device 6900 and the remote user of the remote device.

The second stage 6925 illustrates the user starting a snap-to-corneroperation by selecting an inset display area 6910. In this example, aselection is made by placing a finger 6950 anywhere within the insetdisplay area 6910. As shown, this selection is displayed in terms of athick border 6962 for the inset display 6910. Different embodiments mayindicate such a selection in different ways, such as by highlighting theinset display 6910, by causing the inset display 6910 to vibrate, etc.

The third stage 6930 illustrates the UI 6960 after the user begins tomove the inset display area 6910 of the PIP display 6965 from one areain the PIP display 6965 to another area in this PIP display 6965. Inthis example, the inset display area 6910 has started to move from thelower right corner of the PIP display 6965 to the upper right corner ofthis display, as indicated by the arrow 6955. The inset display 6910 ismoved by the user dragging his finger 6950 towards the upper rightcorner of the PIP display 6965 after selecting the inset display 6910.Some embodiments provide other techniques for moving the inset display6910 around in the PIP display 6965.

The fourth stage 6935 illustrates the UI 6960 in a state after the userhas removed his finger 6950 from the screen of the device 6900. In thisstate, the inset display area 6910 is still moving towards the upperright corner of the PIP display 6965 that was identified based on theuser's finger movement in the third stage. In other words, after thefinger 6950 starts the movement of the inset display 6910 towards theupper right corner of the PIP display 6965, the UI 6960 maintains thismovement even after the finger 6950 is removed. To maintain thismovement, the UI 6960 of some embodiments require the user's dragoperation to be larger than a particular threshold amount (e.g., longerthan a particular distance or longer than a particular length of time)before the user removes his finger; otherwise, these embodiments keepthe inset display area in its original bottom right corner positionafter moving this display area slightly or not moving it at all.

However, while some embodiments allow the inset display area to moveeven after the user stops his drag operation before the inset displayarea has reached its new location, other embodiments require the user tomaintain his drag operation until the inset display area reaches its newlocation. Some embodiments provide still other techniques for moving theinset display area. For example, some embodiments may require the userto specify where to direct the display area 6910 before the display area6910 actually starts to move, etc. Some embodiments may also allowdisplay areas to slide and snap-to-corners by simply tilting the mobiledevice in different angles.

The fifth stage 6940 illustrates the UI 6960 after the inset displayarea 6910 has reached its new location at the upper right corner of thePIP display area 6965. The removal of the thick border 6962 in the fifthstage indicates that the snap-to-corner operation is completed.

To facilitate the movement illustrated in the above-described third,fourth and fifth stages 6930, 6935, and 6940, the UI 6960 of someembodiments employs snapping rules that allow the inset display area6910 to quickly snap to a corner of the PIP display 6965 once the usercauses the inset display area 6910 to move towards that corner. Forinstance, when the user drags the inset display area 6910 by more than athreshold amount towards a particular corner, the UI 6960 of someembodiments identifies the direction of motion of the inset display6910, determines that the motion has exceeded a threshold amount, andthen subsequently moves the inset display area 6910 automaticallywithout further user input to the next grid point in the UI 6960 towhich the inset display 6910 can be snapped. In some embodiments, theonly grid points that are provided for snapping the inset display 6910are grid points at the four corners of the PIP display 6965. Otherembodiments provide other grid points in the UI 6960 (e.g., in the PIPdisplay 6965) to which the inset display 6910 can snap.

Still other embodiments may not employ grid points so that the insetdisplay area 6910 can be positioned at any point in the PIP display. Yetother embodiments provide a feature that allows the user to turn on oroff the snap to grid point feature of the UI. Moreover, in addition tothe video captured from the devices, different embodiments may allow theuser to perform the snap-to-corner operations to various items, such asicons, etc. As noted above, the moving of a display area(s) of acomposite display can cause changes to the image processing operationsof the dual camera mobile device such as causing the video conferencemanager 1604 to re-composite the display area in the composite displayin response to the user's input.

5. Push and Snap

The example in FIG. 69 illustrates a snap-to-corner operation thatallows a user of a dual camera mobile device to move one of two insetdisplay areas from one corner of the PIP display to another corner thatis not occupied by an inset display. Some embodiments enable a pushfeature that moves a first inset to a location of a second inset andalso pushes the second inset to a new location. FIG. 70 illustrates onesuch example that is performed during a video conference.

FIG. 70 illustrates the movement of an inset display from one corner ofthe PIP display to another corner of the PIP display that is occupied byanother inset display, by reference to six different stages 7020, 7025,7030, 7035, 7040 and 7045 of this UI 6960. The first stage 7020illustrates the UI 6960 during a video conference between a local userof the device and a remote user of a remote device. The UI 6960 in FIG.70 shows a PIP display 6965 that is the same PIP display shown in thefirst stage of FIG. 69 after the video conference has started. In thisexample, the video captured by the local user's device is displayed inthe inset display areas 6905 and 6910 and the video captured by theremote user's device is displayed in the background display area 6915.

The second stage 7025 illustrates the user starting the snap-to-corneroperation by selecting inset display area 6905. In this example, aselection is made by placing a finger 7055 anywhere within the insetdisplay area 6905. As shown, this selection is displayed in terms of athick border 7065 for the inset display 6905. Different embodiments mayindicate such a selection in different ways, such as by highlighting thedisplay area 6905, by causing the display area 6905 to vibrate, etc.

The third stage 7030 illustrates the UI 6960 after the user begins tomove the inset display area 6905 from the lower left corner of the PIPdisplay 6965 to the lower right corner of the PIP display 6965 (bydragging his finger 7055 towards the lower right corner of the PIPdisplay 6965 after selecting the inset display 6905 in the third stage),as indicated by the arrow 7050. Some embodiments provide othertechniques for moving the inset display area 6905 around in the PIPdisplay 6965.

The fourth stage 7035 illustrates the UI 6960 after the inset displayarea 6905 has come in contact with the inset display area 6910. Uponcontact, the inset display area 6910 moves towards the next nearestcorner. In this example, the inset display area 6910 starts to move inthe direction (as indicated by arrow 7075) of the upper right corner ofthe PIP display 6965. The activation of this push operation is displayedin terms of a thick border 7070 for the inset display 6910. Differentembodiments may indicate such an activation in different ways, such asby highlighting the display 6910, etc.

The fifth stage 7040 illustrates the UI n in a state after the insetdisplay area 6905 has snapped to the lower right corner that waspreviously occupied by the inset display area 6910. In this state, theinset display area is still moving towards the upper right corner of thePIP display 6965. Also, the thick border 7065 is no longer displayed. Solong as the user's drag operation from the third stage 7030 is largerthan a threshold that causes the inset display area 6905 to snap to theright corner, the inset display area 6910 is removed from its corner andsnaps all the way to the next nearest corner.

Some embodiments include a set of rules for determining which way topush the second inset display area 6910. In the case illustrated in FIG.70 , some embodiments attempt to continue the rotation of the insetdisplay areas. That is, because display area 6905 is moved in acounterclockwise direction, the display area 6910 is also movedcounterclockwise. Some embodiments provide a hierarchy of possiblelocations to which the pushed inset display area 6910 can move andselect the first unoccupied location on the list. For example, the topright corner might be the first location in such a list when an insetdisplay area in the bottom right is pushed by an inset display areacoming from the bottom left. If, however, a third inset display area wasalready present in the upper right corner, some embodiments would moveto the next option on the list (e.g., the upper left corner, the center,or the lower left corner). Other embodiments would push the third insetdisplay area with the second inset display area such that the devicewould need to determine a new location for the third inset display area.

The sixth stage 7045 illustrates the UI 6960 after the inset displayarea 6910 has reached its new location at the upper right corner of thePIP display area 6965. The removal of the thick border 7070 in thisstage indicates that the snap-to-corner-push operation is completed.Similar to the push-to-corner operation described by reference to FIG.68 , the moving of a display area(s) of a composite display can causechanges to the image processing operations of the dual camera mobiledevice such as causing the video conference manager 1604 to re-compositethe display area in the composite display in response to the user'sinput.

6. Rotate

Some embodiments rotate a PIP display that is presented during a videoconference when a user of a mobile device used for the video conferencerotates the device during the conference. FIG. 71 illustrates therotation of a UI display 7175 of a device 7100 when the device isrotated from a vertical position to a horizontal position. The device7100 is held vertically when the long side of the screen is vertical,whereas the device 7100 is held horizontally when the long side of thescreen is horizontal. In the example illustrated in FIG. 71 , the UIdisplay 7175 rotates from a portrait view that is optimized for avertical holding of the device to a landscape view that is optimized forhorizontal holding of the device 7100. This rotation functionalityallows the user to view the UI 7175 displayed in an upright positionwhen the mobile device 7100 is held either vertically or horizontally.This example illustrated in FIG. 71 is similar to the exampleillustrated in FIG. 34 , except that FIG. 71 illustrates rotating a PIPdisplay that includes two inset display areas rather than only one.

In FIG. 71 , the UI 7175 of the mobile device 7100 presents the PIPdisplay 7180 during a video conference with a remote user of anothermobile device. The PIP display 7180 in FIG. 71 includes three videodisplays: a background main display 7115 and two foreground insetdisplays 7110 and 7160. In this example, the background main display7115 presents a video of a mountain, which is assumed to be videocaptured by the remote device's front camera or back camera. Theforeground inset display 7110 presents a video of a smiley face in aroom, which is assumed to be captured by the local device's front cameraor back camera. The other foreground inset display 7160 presents a videoof a guitarist singing, which is assumed to be a guitarist whose videois being captured by the other camera of the local device. Below the PIPdisplay 7180 is an End Conference button 7155, which the user may selectto end the video conference (e.g., through a single finger tap). ThisPIP display is only one manner of presenting a composite view of thevideos being captured by the remote and local devices. Some embodimentsmay provide other composite views, such as tiled views or different PIPdisplays.

FIG. 71 illustrates the rotation of the UI 7175 in terms of sixdifferent operational stages 7120, 7125, 7130, 7135, 7140 and 7145. Thefirst stage 7120 illustrates the UI 7175 during a video conferencebetween the local user of the device and the remote user of the remotedevice.

The second stage 7125 illustrates the UI 7175 after the user begins totilt the device 7100 sideways. In this example, the device 7100 hasstarted to tilt the device 7100 from being held vertically to being heldhorizontally, as indicated by the arrow 7185. The appearance of the UI7175 has not changed. In other situations, the user may want to tilt thedevice 7100 from being held horizontally to being held verticallyinstead, and in these situations the UI display 7175 switches from ahorizontally optimized view to a vertically optimized view.

The third stage 7130 illustrates the UI 7175 in a state after the device7100 has been tilted from being held vertically to being heldhorizontally. In this state, the appearance of the UI display 7175 stillhas not changed. In some embodiments, the rotation operation istriggered after the device 7100 is tilted past a threshold amount and iskept past this point for a duration of time. In the example illustratedin FIG. 71 , it is assumed that the threshold amount and the speed ofthe rotation do not cause the UI display 7175 to rotate until a shorttime interval after the device has been placed in the horizontalposition. Different embodiments have different threshold amounts andwaiting periods for triggering the rotation operation. For example, someembodiments may have such a low threshold to triggering the rotationoperation as to make the UI 7175 appear as if it were always displayedin an upright position, notwithstanding the orientation of the device7100. In other embodiments, the user of the device 7100 may specify whenthe rotation operation may be triggered (e.g., through a menu preferencesetting). Also, some embodiments may not delay the rotation after thedevice is tilted past the threshold amount. Moreover, differentembodiments may allow the rotation operation to be triggered indifferent ways, such as by toggling a switch on the mobile device, bygiving voice commands, upon selection through a menu, etc.

The fourth stage 7135 illustrates the UI 7175 after the rotationoperation has started. Some embodiments animate the rotation displayareas to provide feedback to the user regarding the rotation operation.FIG. 71 illustrates an example of one such animation. Specifically, itshows in its fourth stage 7135 the start of the rotation of the displayareas 7110, 7115, and 7160 together. The display areas 7110, 7115, and7160 rotate around an axis 7165 going through the center of the UIdisplay 7175 (i.e., the z-axis). The display areas 7110, 7115, and 7160are rotated the same amount but in the opposite direction of therotation of the device 7100 (e.g., through the tilting of the device7100). In this example, since the device 7100 has rotated ninety degreesin a clockwise direction (by going from being held vertically to beingheld horizontally) the rotation operation would cause the display areas7110, 7115, and 7160 to rotate ninety degrees in a counter clockwisedirection. As the display areas 7110, 7115 and 7160 rotate, the displayareas 7110, 7115, and 7160 shrink proportionally to fit the UI display7175 so that the display areas 7110, 7115, and 7160 may still appearentirely within the UI 7175. Some embodiments may provide a message toindicate the state of this device 7100 (e.g., by displaying the words“Rotating”).

The fifth stage 7140 illustrates the UI 7175 after the display areas7110, 7115, and 7160 have rotated ninety degrees counter clockwise fromportrait view to landscape view. In this stage, the display areas 7110,7115, and 7160 have been rotated but have not yet expanded across thefull width of the UI 7175. The arrows 7170 indicate that at the end ofthe fifth stage, the display areas 7110, 7115, and 7160 will start tolaterally expand to fit the full width of the UI 7175. Differentembodiments may not include this stage since the expansion could beperformed simultaneously with the rotation in the fourth stage 7135.

The sixth stage 7145 illustrates the UI 7175 after the display areas7110, 7115 and 7160 have been expanded to occupy the full display of theUI 7175. As mentioned above, other embodiments may implement thisrotation differently. For some embodiments, simply rotating the screenof a device past a threshold amount may trigger the rotation operation,notwithstanding the orientation of the device 7100.

Also, other embodiments might provide a different animation forindicating the rotation operation. The rotation operation performed inFIG. 71 involves the UI display 7175 rotating about the center of the UIdisplay 7175. Alternatively, the display areas may be individuallyrotated about the center axis of their individual display areas. Onesuch approach is shown in FIG. 72 , which shows an alternative method toanimating the rotation of the PIP display area 7180 of the UI 7175. ThePIP display illustrated in FIG. 72 is the same PIP display 7180illustrated in FIG. 71 .

FIG. 72 illustrates the rotation of the PIP display 7180 in terms of sixdifferent operational stages 7120, 7125, 7130, 7220, 7225, and 7230. Thefirst three stages of operation of the UI 7175 are identical to thefirst three stages of operation as described in the UI 7175 in FIG. 71 .At the third stage for both FIGS. 71 and 72 , the device has gone frombeing held vertically to being held horizontally and the rotation of theUI 7175 has not yet begun.

The fourth stage 7220 illustrates an alternative method to animating therotation. In this stage, the rotation operation has started.Specifically, the fourth stage 7220 shows the start of the rotation ofthe display areas 7110, 7115, and 7160. The display areas 7110, 7115,and 7160 each rotate around axes 7250 going through the center of eachof the display areas (i.e., the z-axis). The display areas 7110, 7115,and 7160 are rotated the same amount but in the opposite direction ofthe rotation of the device 7100 (e.g., through the tilting of the device7100). In this example, since the device 7100 has rotated ninety degreesin a clockwise direction (by going from being held vertically to beingheld horizontally) the rotation operation would cause the display areas7115, 7110 and 7160 to rotate ninety degrees in a counter clockwisedirection. As the display areas 7115, 7110 and 7160 rotate, they alsoshrink proportionally to fit the UI display 7175 so that the displayareas 7115, 7110 and 7160 may still appear entirely on the UI 7175.

The fifth stage 7225 illustrates the UI 7175 after the display areas7115, 7110 and 7160 have rotated ninety degrees counter clockwise fromportrait view to landscape view. In this stage, the display areas 7115,7110 and 7160 have been rotated but have not yet expanded across thefull width of the UI 7175 or reached their final positions. The finalpositions of the display areas in the PIP display 7115 are determined bythe positions of the display areas in the PIP display as shown in thefirst stage 7120 (e.g., the inset display 7110 in the lower left cornerand the inset display 7160 in the lower right corner of the PIP display7180).

The arrows 7170 indicate that at the end of the fifth stage, the displayareas 7115, 7110 and 7160 will start to laterally expand until maindisplay area 7115 fits the full width of the UI 7175. Moreover, thearrow 7255 indicates that the inset display areas 7110 and 7160 willmove to reach their final positions in the PIP display 7180. In otherwords, the inset display area 7110 will move down towards the lower leftcorner of the PIP display 7180 while the other inset display area 7160moves to the lower right corner of the PIP display 7180. Differentembodiments may perform this animation differently, e.g. by using thesnap and push operation illustrated in FIG. 71 . The sixth stage 7230illustrates the UI 7175 after the display areas 7110, 7115, and 7160have been expanded to occupy the full display of the UI 7175 and havemoved to their final positions.

As mentioned above, other embodiments may implement this rotationdifferently. For instance, as similarly illustrated in FIGS. 36 and 37 ,some embodiments provide a rotation operation in which the orientationof the display area that displays the video captured by the local devicechanges to reflect the orientation of the local device after a rotationoperation is performed on the local device, some embodiments provide arotation operation in which the orientation of the display area thatdisplays video captured by the remote device changes to reflect theorientation of the remote device after a rotation operation is performedon the remote device, some embodiments provide a rotation operation inwhich the display area 1155 remains in the same position, someembodiments provide a different layout in the display area (e.g., thelayout of the display area 1155 of FIG. 12 ), or a combination thereof.

For some embodiments, simply rotating the screen of a device past athreshold amount may trigger the rotation operation, notwithstanding theorientation of the device 7100. As also mentioned above, the local andremote devices notify each other of rotate operations performed on oneof the devices through a control communication channel in order to allowthe other device to perform any corresponding modifications to the onedevice's video. Also, the animation of the rotation operation can causechanges to the operation of the cameras or the image processingoperations of the dual camera mobile device such as causing the videoconference manager 1604 to re-composite the display area(s) at differentangles in the UI 1105 and to scale the images displayed in the displayarea(s).

7. Select Remote View to View

As mentioned above, some embodiments allow a user of a dual cameramobile device to select which camera to use for a video conferencebefore the start of the video conference or at the start of the videoconference. Instead of, or in conjunction with this capability, someembodiments allow the user of the device to select between two videosthat are being displayed in the video conference and that either arefrom two video cameras of a remote device or are from two video camerasof the user's local device. FIG. 73 illustrates an in-conferenceselection of one video from two remote videos, while FIG. 74 illustratesan in-conference selection of one video from two local videos.

FIG. 73 illustrates the selection of the remote video in terms of sixoperational stages 7335, 7340, 7345, 7350, 7355, and 7360 of a UI 7375displayed on a local device 7300. The first stage 7335 illustrates theUI 7375 with an initial PIP display 7390 that is being presented duringa video conference with a remote user of a mobile device that has twocameras.

As shown in the first stage 7335, the initial PIP display 7390 includesthree displays: a background main display 7315 and two foreground insetdisplays 7305 and 7310. The background display 7315 takes a majority ofthe PIP display area 7390, while the foreground inset displays 7305 and7310 overlap portions of the background display 7315 on the UI 7375. Inthis example, the background display 7315 presents a video of a personin front of a microphone, which is assumed to be video captured by theremote device's back camera. The first foreground inset display 7305presents a video of a man's face, which in this example is assumed to bevideo captured by one of the cameras of the local device 7300. Thesecond foreground inset display 7310 presents a video of a person with ahat, which in this example is assumed to be video captured by the remotedevice's front camera.

The initial PIP display 7390 is only one manner of presenting acomposite view of the videos being captured by the cameras of the localdevice and remote device. Some embodiments may provide other compositeviews. For instance, the background display may present the video fromone of the local device's cameras, and the smaller foreground insetdisplays may present the videos from the remote device's front and backcamera. Also, in some cases, the PIP display only includes onebackground video display and one foreground video display, both of whichcome from the remote device. The manner of the PIP display or a defaultdisplay mode may be specified by the user in some embodiments.

The second stage 7340 illustrates the start of the video selectionoperation. In this example, this operation is initiated by invoking aset of selectable UI items to be displayed on the PIP display 7390. Theset of selectable UI items present options for selecting the remotevideo for display. In some embodiments, the set of selectable UI itemsmay be invoked by selecting (e.g., by touching) any display area that isplaying a remote video on the UI 7375. In other embodiments, the itemsmay be invoked by selecting (e.g., by touching) anywhere on UI 7375.Instead of or in conjunction with such invocation operations, someembodiments also allow the user to invoke the set of selectable UI itemsthrough other operations, such as through different touchscreenoperations or using one or more other physical inputs of the device.

The third stage 7345 displays the UI 7375 with the invoked set ofselectable UI items 7380 for selecting the remote videos. In thisexample, the set of selectable UI items 7380 in the form of a pop-upmenu is displayed in the PIP display area 7390 and it overlaps the PIPdisplay. The set of selectable UI items 7380 (which can be implementedas selectable buttons) includes a “Select R1” selectable UI item 7320(e.g. button 7320), a “Select R2” selectable UI item 7325 (e.g. button7325), a “Select Both” selectable UI item 7330 (e.g. button 7330), and a“Cancel” selectable UI item 7385 (e.g. button 7385). In this example,selection of the “Select R1” button 7320 would cause the UI 7375 todisplay only the video captured by the remote device's back camera(being presented in the background display 7315). Selection of the“Select R2” button 7325 would cause the UI 7375 to display only thevideo captured by the remote device's front camera (being presented inthe foreground inset display 7310). Selection of the “Select Both”button 7330 would cause the UI 7375 to continue displaying both videoscaptured by the remote device's front and back cameras. Selection of the“Cancel” button 7385 would cancel the operation. In some embodiments,the video captured by the local device is not affected by the selectionmade on this menu.

The fourth stage 7350 illustrates the UI 7375 after the user hasselected the “Select R1” button 7320 (e.g., by tapping on the button7320 with his finger 7365). This selection is indicated by thehighlighting of the button 7320 on the UI 7375. Some embodiments usedifferent indication displays (e.g., highlighting the border of theselected item or the text in the selected item).

The fifth stage 7355 illustrates the animation of the UI 7375 after theuser has selected the video from R1 for display. In this example, the UI7375 removes the unwanted foreground inset display area 7310 by slidingit off the right edge of the PIP display 7390 as indicated by arrows7370. Other embodiments utilize different animations to remove theunwanted inset display area, such as fading out or dissolving the inset,moving it in a different direction, or simply removing itinstantaneously.

The sixth stage 7360 displays the UI 7375 during the video conferenceafter the video selection operation has been completed. The videodisplay area 7310 is no longer displayed on the UI 7375. In this stage,the UI 7375 presents a new PIP display 7395 that includes the videodisplay area 7315 as the background main display and the video displayarea 7305 as the inset display.

In some embodiments, this video selection operation will also cause theremote device to only display the selected video, though in otherembodiments the operation has no effect on the remote device. In someembodiments, this video selection operation will cause the remote deviceto stop transmitting the unwanted video to the local device. In fact,this video selection operation will cause the camera of the remotedevice to stop capturing the unwanted video in some embodiments. In someembodiments, these effects on the remote device can be overruled by theuser of the remote device.

The above example illustrates the case where the remote view selected isthat which is already displayed in the background main display. In someembodiments, when the user selects the remote view that is displayed inone of the inset displays, the selected remote view is displayed in thebackground main display. Some such embodiments use an animation likethat displayed in FIG. 68 in this case. Moreover, the selection of theremote video(s) can cause changes to the image processing operations ofthe local dual camera mobile device such as causing the video conferencemanager 1604 to composite only the selected remote video(s) in thecomposite display in response to the user's input.

8. Select Local View to View

FIG. 74 illustrates the selection of the local video in terms of sixoperational stages 7435, 7440, 7445, 7450, 7455, and 7460 of a UI 7475displayed on a local device 7400. The first stage 7435 illustrates theUI 7475 with an initial PIP display 7490 that is being presented duringa video conference with a remote user of a mobile device having at leastone camera. The PIP display 7490 is similar to the one in the firststage 7335 in FIG. 73 , except that unlike FIG. 73 , the backgrounddisplay 7415 presents a video of a man that is captured by a remotedevice's camera, the left foreground inset display 7410 presents a videoof a person with a guitar that is captured by the back camera of localmobile device, and the right foreground inset display 7405 presents avideo of a man with a hat that is captured by the front camera of localmobile device 7400. Thus, only one remote video is displayed, while twolocal videos are displayed.

The second stage 7440 illustrates the start of the video selectionoperation. In this example, this operation is initiated by invoking aset of selectable UI items to be displayed on the PIP display 7490 forselecting the remote video for display. In some embodiments, the set ofselectable UI items may be invoked by selecting (e.g., by touching) anydisplay area that is playing a local video on the UI display 7475. Inother embodiments, the items may be invoked by selecting (e.g., bytouching) anywhere on UI display 7475. Instead of, or in conjunctionwith such invocation operations, some embodiments also allow the user toinvoke the set of selectable UI items through other operations, such asthrough different touchscreen operations or using one or more otherphysical inputs of the device.

The third stage 7445 displays the UI 7475 with the invoked set ofselectable UI items 7480 for selecting the local videos. In thisexample, the set of selectable UI items 7480 in the form of a pop-upmenu is displayed in the PIP display area 7490 overlapping the PIPdisplay. The set of selectable UI items 7480 includes a “Select L1”selectable UI item 7420 (e.g. button 7420), a “Select L2” selectable UIitem 7425 (e.g. button 7425), a “Select Both” selectable UI item 7430(e.g. button 7430), and a “Cancel” selectable UI item 7485 (e.g. button7485) for canceling the operation. In this example, selection of the“Select L1” button 7420 would cause the UI 7475 to display only thevideo captured by the local device's back camera (being presented in theforeground inset display 7410). Selection of the “Select L2” button 7425would cause the UI 7475 to display only the video captured by the localdevice's front camera (being presented in the foreground inset display7405). Selection of the “Select Both” button 7430 would cause the UI7475 to continue displaying both videos captured by both cameras on thelocal device, and selecting the “Cancel” button 7485 would cancel theoperation. In some embodiments, the video captured by the remote deviceis not affected by the selection made through this menu.

The fourth stage 7450 illustrates the UI 7475 after the user hasselected the “Select L2” button 7425 (e.g., by tapping on the button7425 with his finger 7465). This selection is indicated by thehighlighting of the button 7425 on the UI display 7475. Some embodimentsuse different indication displays (e.g., highlighting the border of theselected item or the text in the selected item).

The fifth stage 7455 displays the animation of the UI 7475 after theuser has selected the video from L2 for display. In this example, the UI7475 removes the unwanted foreground inset display 7410 by sliding itoff the left edge of the PIP display 7490 as indicated by arrows 7470.Other embodiments utilize different animations to remove the unwantedinset display area, such as fading out or dissolving the inset, movingit in a different direction, or simply removing it instantaneously.

The sixth stage displays the UI 7475 during a video conference after thevideo selection operation has been completed. The video display area7410 is no longer on the UI 7425. In this stage, the UI 7475 presents anew PIP display 7495 that includes the remote video display 7415 as thebackground main display and the local video display 7405 as an insetdisplay. In some embodiments, this video selection operation onlyaffects the local display, as both video captures are still transmittedto the remote device. Other embodiments stop capturing from the removedcamera.

The above example illustrates the case where the local view selected isthat which is already displayed in the background main display. In someembodiments, when the user selects the local view that is displayed inone of the inset displays, the selected local view is displayed in themain background display. Some such embodiments use an animation likethat displayed in FIG. 68 in this case. Other embodiments will use aninset remote view when the local view in the background main display isremoved.

Similar to the remote view selection operation described above byreference to FIG. 73 , the selection of the local video(s) can causechanges to the image processing operations of the local dual cameramobile device such as causing the video conference manager 1604 to thecomposite only the selected remote video(s) in the composite display inresponse to the user's input. The selection of the local video(s) canalso cause changes in the operation of a camera(s) of the local device.For example, some embodiments cause the camera of an unselected video tostop transmitting the unwanted video to the remote device while otherembodiments cause the camera to stop capturing the unwanted video.

9. Select Local View to Transmit

The above sub-sections illustrate in-conference modifications to thevideo displays. Some embodiments also allow a user of a dual cameramobile device to select which camera to use for a video conferencebefore the start of the video conference. FIG. 75 illustrates apre-conference selection of one video from two videos captured by theuser's dual camera mobile device to use for the video conference.

FIG. 75 illustrates the selection of the local video to use for thevideo conference in terms of eight operational stages of a UI 7500. Thefirst stage 7502 illustrates the UI 7500 of a dual camera mobile device7518 with an initial PIP display 7542 that is being presented after auser has requested to start a video conference with a remote user of amobile device.

As shown in the first stage 7502, the initial PIP display 7542 includestwo video displays: a background main display 7520 and a foregroundinset display 7522. The background main display 7520 takes up a majorityof the display screen of the device, while the foreground inset display7522 is smaller and overlaps the background main display. In thisexample, the background display 7520 presents a video of a personholding a guitar, which is assumed to be video being captured by thedevice's back camera. The foreground inset display 7522 presents a videoof a person with a hat, which in this example is assumed to be videobeing captured by the device's front camera.

This initial PIP display 7542 is only one manner of presenting acomposite view of the videos being captured by the cameras of the localdevice. Some embodiments may provide other composite views. Forinstance, the background display may present the video from the device'sfront camera, and the smaller foreground inset display may present thevideo from the device's back camera. Also, some embodiments allow thetwo videos to appear in the UI 7500 in two side-by-side display areas(e.g. left and right display windows, or top and bottom display windows)or two diagonally aligned display areas. The manner of the PIP displayor a default display mode may be specified by the user in someembodiments. Below the PIP display is a selectable UI item 7540 labeled“End Conference” (e.g. a button 7540) that allows the user to end thevideo conference by selecting the item.

In the first stage 7502, the user of the mobile device 7518 hasrequested to have a video conference with a remote user and is waitingfor the remote user to respond. This waiting period is illustrated bythe “Preview, Waiting for response . . . ” notation at the bottom of thedisplay.

The second stage 7504 illustrates the start of the video selectionoperation. In this example, the operation is initiated by invoking a setof selectable UI items to be displayed on the PIP display 7542. The setof selectable UI items present options for selecting the local video totransmit to the remote device for the video conference. In someembodiments, the set of selectable UI items may be invoked by selecting(e.g., touching) anywhere on the UI display 7500 during thepre-conference time while waiting for the remote user to respond.Instead of, or in conjunction with such invocation operations, someembodiments also allow the user to invoke the set of selectable UI itemsthrough other operations, such as through different touchscreenoperations or using one or more other physical inputs of the device.

The third stage 7506 illustrates the UI 7500 with the invoked set ofselectable UI items 7526 for the user to select the videos. In thisexample, the set of selectable UI items 7526 in the form of a pop-upmenu is displayed in the PIP display area 7542 and it overlaps the PIPdisplay. In this example, the set of selectable UI items includes: a“Transmit L1” item 7528 (e.g. button 7528); a “Transmit L2” item 7530(e.g. button 7530); a “Transmit Both” item 7532 (e.g. button 7532); anda “Cancel” item 7534 (e.g. button 7534). In this example, selection ofthe “Transmit L1” button 7528 would cause the UI 7500 to transmit onlythe video captured by the device's back camera to the remote deviceduring the video conference. Selection of the “Transmit L2” button 7530would cause the UI 7500 to transmit only the video captured by thedevice's front camera to the remote device during the video conference.Selection of the “Transmit Both” button 7532 would cause the UI 7500 totransmit both videos captured by the device's front and back camera tothe remote user for the video conference, and selecting the “Cancel”button 7534 would cancel the operation.

The fourth stage 7508 illustrates the UI 7500 after the user hasselected the “Transmit L1” button 7528 (e.g., by tapping on the button7528 with his finger 7524). This selection is indicated by thehighlighting of the button 7528 on the PIP display area 7542. Someembodiments use different indication displays (e.g., highlighting theborder of the selected item or the text in the selected item).

The fifth stage 7510 illustrates the animation of the UI 7500 after theuser has selected the video from the device's back camera to transmit tothe remote device. In this example, the UI 7500 removes the unwantedforeground inset display 7522 by sliding it off the right edge of thePIP display 7542 as indicated by arrows 7536. In the sixth stage 7512,the inset display 7522 has been completely removed from the PIP displayarea 7542. Different embodiments use different animations to remove theunwanted display area, such as fading out or dissolving the displayarea, moving it in a different direction, or simply removing itinstantaneously.

The seventh stage 7514 illustrates the animation of the UI 7500 afterthe remote user has accepted the video conference request. Theacceptance of the video conference request is highlighted by the removalof the “Preview, Waiting for response . . . ” notation on the display.In this stage, the background display area 7520, which is a video fromthe device's back camera, gradually decreases in size to the lower leftcorner of the PIP display area 7542, as indicated by arrows 7538. Thebackground display 7520 shrinks so that the UI 7500 can show behind thedisplay area 7520 a display area 7544 that contains the video from acamera of the remote user. Some embodiments shrink the local camera to adifferent location, use a tiled composite display of the two displayedvideos, or make the remote view the inset display area of a PIP display.

The eighth stage 7516 shows the UI 7500 after the video selectionoperation has been completed. The UI 7500 presents a new PIP display7546 that includes an inset display 7520 of the video captured from thelocal device and a background display 7544 of the video transmitted fromthe remote device.

B. Bandwidth & Frame Rate

In some embodiments, adjustments to the size of the display areas of aremote mobile device during a video conference may cause a local mobiledevice to reallocate the bandwidth allotted to each video captured bythe two cameras (i.e., a front camera and a back camera) of the localmobile device. FIG. 76 illustrates two examples of such bandwidthreallocation between the two cameras of the local device.

Each of the examples in FIG. 76 involves a back camera sensor 7605 of alocal device, a front camera sensor 7610 of the local device, a videoconference module 7615 of the local device, and a UI 7635 of a remotemobile device 7620. The back camera sensor 7605 and the front camerasensor 7610 capture videos from the respective back and front cameras ofthe local device. The captured videos are sent to the video conferencemodule 7615, which processes them and transmits them to the remotedevice for display in the UI 7635.

In FIG. 76 , the UI 7635 of the remote device presents a compositedisplay. The composite display shows videos that are captured by thelocal device's front and back camera. The video from the front cameracaptures a tree and a man with a hat while the video from the backcamera captures a mountain landscape. As illustrated in FIG. 76 , thetwo videos may be displayed in the UI 7635 in many different mannersbased on the arrangement of display areas for displaying the videos andalso the size of the display areas. In each example, the videoconference module 7615 initially allocates the total output bandwidthbetween each of the videos according to the relative sizes of thedisplay areas in the remote device. Specifically, the video that isdisplayed in a larger display area in the UI 7635 is allocated a largerportion of the total bandwidth, and the video that is displayed in asmaller display area in the UI 7635 is allocated a smaller portion ofthe bandwidth. In some embodiments, when the videos are displayed in thesame size display area, the total output bandwidth is allocated equallybetween the two videos.

The amount of bandwidth allocated to each of the two videos may impactthe manner in which each video is processed. For example, a video mayrequire a higher bandwidth than what is allocated to the video. In suchcase, the video's frame rate is adjusted or the size of the video'simages is scaled down in order to accommodate the lower bandwidth.Reducing the frame rate of a video causes the video to appear “choppy”whereas scaling down the size of the video's images reduces the area inwhich the video is displayed. Therefore, when a video is allocated anamount of bandwidth, some embodiments adjust the video's frame rate,scale down the size of the video's images, or perform a combination ofboth in order to ensure that the video can be transmitted within theallotted bandwidth. One of ordinary skill in the art will realize thatthe adjustment of frame rate and average frame size may vary to obtainan optimal overall video quality while still ensure that the video canbe transmitted within the allotted bandwidth.

Example (1) of FIG. 76 illustrates one scenario of bandwidthreallocation in two operational stages of UI 7635. The UI 7635 of theremote device 7620 in first stage 7670 presents a composite display thatcontains two displays—one on the top and the other on the bottom of theUI 7635. In this example, the top display area 7625 shows a video thatis being captured by the local device's front camera, and the bottomdisplay area 7630 shows a video that is being captured by the localdevice's back camera. As illustrated in the first stage 7670, the topdisplay area 7625 is larger than the bottom display area 7630. Thus, thevideo from the local device's front camera is allocated 80% of thebandwidth, and the video from the local device's back camera isallocated 20% of the bandwidth. In order to ensure that the video fromthe local device's back camera can be transmitted from the local deviceto the remote device within the allotted bandwidth, the video's framerate or scaling size, or both, are adjusted.

The second stage 7675 illustrates the UI 7635 after the user of theremote device has increased the size of the bottom display area suchthat the sizes of the top display area 7625 and the bottom display area7630 are approximately the same. As a result, each of the videos isreallocated 50% of the total bandwidth by the video conference module7615.

Example (2) of FIG. 76 illustrates another scenario for bandwidthreallocation in two operational stages of UI 7635. In the first stage7680 of Example (2), the UI 7635 presents a PIP display. The PIP displaycontains two displays: a background main display area 7650 and aforeground inset display area 7655. The background display area 7650takes up a majority of the PIP display, while the foreground insetdisplay area 7655 is smaller and overlaps the background main displayarea 7650. In this example, the background display area 7650 presents avideo that is being captured by the device's front camera. The insetdisplay area 7655 presents a video that is being captured by thedevice's back camera. As illustrated in this stage, the backgrounddisplay area 7650 is larger than the inset display area 7655. Thus, thevideo from the device's front camera is allocated 80% of the bandwidth,and the video from the device's back camera is allocated 20% of thebandwidth. In order to ensure that the video from the local device'sback camera can be transmitted from the local device to the remotedevice within the allotted bandwidth, the video's frame rate or scalingsize, or both, are adjusted.

The second stage 7685 illustrates the UI 7635 after the user of theremote device has swapped the displays of the two videos. Specifically,the background display area 7660 now presents a video that is beingcaptured by the device's back camera, and the inset display area 7665now presents a video that is being captured by the device's frontcamera. Since the sizes of the display areas for the two videos havechanged, the video from the device's back camera is allocated 80% of thebandwidth, and the video from the device's front camera is allocated 20%of the bandwidth. As such, the frame rate or scaling size, or both, ofthe video from the local device's front camera will be reduced. One ofordinary skill in the art will realize that the bandwidth distributionsdescribed in FIG. 76 are only examples and other techniques forallocating bandwidth between two cameras during a video conference arepossible.

1. Frame Rate Control

Like the in-conference frame rate control operations described above,some embodiments may wish to separately adjust or maintain the rate atwhich images of a video captured by each camera of the dual cameramobile device are transmitted to the other device in the videoconference. Some of these embodiments provide similar techniquesdescribed above. For example, some embodiments control the frame rate ofeach camera by adjusting the VBI of the sensor module 415 of eachcamera. Other embodiments provide additional techniques as well such asframe dropping, which can be performed by the sensor module 415 of eachcamera and/or the universal transmission buffer 3120, for example.

2. Bandwidth Control Through Scaling

As discussed above, during a video conference between a dual cameramobile device and another device, an amount of image data that can betransmitted over one or more network connections in a particular amountof time (i.e., network connection bandwidth) may be limited. To maximizeand maintain throughput of the network connection, different embodimentsof the dual camera mobile device provide different ways to control theamount of image data transmitted over the network connection in theparticular amount of time. In some embodiments, throughput is theaverage rate of successful message delivery over a communication channel(e.g., a network connection).

When transmitting images captured by both cameras of the dual cameramobile device, one such way resizes images from one or both cameras ofthe dual camera mobile device to control the amount of image datatransmitted over the network connection. Some embodiments scale down thesize of the images captured by the dual camera mobile device to reducethe amount of image data transmitted over the network connection whileother embodiments scale up the size of the images to increase the amountof image data transmitted over the network connection.

Some embodiments maintain the height-to-width ratio of the images whenscaling (i.e., uniform scaling). Other embodiments scale the images suchthat the height-to-width ratio of the scaled images are different thatthe original images (i.e., anamorphic scaling).

Furthermore, scaling can be performed at different stages of the imageprocessing process. The scaling of some embodiments can be performed bythe camera sensor. In such embodiments, the camera sensor may drop rowsor columns of data of an image (i.e., pixel values). In some of suchembodiments, the remaining image data is interpolated to smooth theappearance of the image.

The scaling of other embodiments is performed by the scaler module 455of the CIPU 400. In some embodiments, scaling is performed by the videoconference manager 1604, as described above, and in other embodiments,scaler is performed by the encoder. As such, different embodiments ofthe dual camera mobile device perform scaling differently.

3. Bit Rate Control

Some embodiments provide different mechanism for managing the bit rateat which videos captured by the cameras of a dual camera mobile deviceare encoded. In some embodiments, the dual camera mobile device includesa rate controller for each camera. Some embodiments provide a fixed bitrate management scheme. Under this scheme, each of the rate controllersis set at a fixed bit rate so that the total bit rate of the videos fromboth cameras on the mobile device is constant. Other embodiments providea priority scheme in which one of the two videos from the device'scameras will always get priority over the other when the total bit rateis required to be reduced.

In some embodiments, an arbitrator module manages the two ratecontrollers of the two cameras. FIG. 77 illustrates an example of sucharbitrator module. As shown in FIG. 77 , a rate controller 7700 sets thebit rate for the front camera and a rate controller 7705 sets the bitrate of the back camera. The rate controllers send the images from thecamera sensors to the encoder 7715. An arbitrator module 7710 isconnected to both rate controllers and controls the setting of the bitrate for each rate controller 7700 and 7705 any number of ways based oninformation such as the available bandwidth, video size for each of thetwo videos, etc., to ensure that both videos can be transmitted to aremote device under the available bandwidth. In addition, the arbitrator7710 can be configured to implement the fixed rate scheme or thepriority scheme mentioned above.

In some other embodiments, the two rate controllers for the two camerascan communicate with each other. Under this scheme, the rate controllerscan exchange information of their respective videos and set the bitrates of the videos accordingly. Several examples of rate controllerrate management mechanisms are provided. Many other differentmechanisms, however, are possible.

4. Video Processing

Some embodiments of the dual camera mobile device process imagescaptured by both cameras of the dual camera mobile device differently indifferent situations. For example, when processing a PIP composite imagethat includes images captured by both cameras of the dual camera mobiledevice, some embodiments selectively perform the TNR process 2000 on thePIP composite image. Some of these embodiments perform the TNR process2000 on only the main image in the PIP composite image while other ofthese embodiments perform the TNR process 2000 on only the inset imagein the PIP composite image.

As another example of processing images captured by both cameras of themobile device, some embodiments scale images captured by both cameras ofthe dual camera mobile device based on various changes to the videoconference such as user adjustments to the display areas (e.g.,enlarging inset of a PIP display, defining a region of interest in adisplayed video, swapping main/inset of a PIP display, etc.) thatdisplay videos, changes to total available bandwidth, etc. Some of theseembodiments scale the images in the manners described above. That is,the images can be scaled by the encoder 1655, the video conferencemanager 1604, the scaler module 455, and the camera sensors (i.e., 405 aor 405 b) by which the images were captured, for example.

5. Encoding

As mentioned above, some embodiments transmit video from both cameras ofa dual camera mobile device. As such, these embodiments may encode thevideos captured by both cameras for transmission to a remote deviceduring a video conference. Different embodiments provide different waysto encode the videos for transmission. FIG. 78 illustrates an approachthat uses a multiplexer (MUX) 7815, an encoder module 7825, a buffer7830 and a combining module 7835 to process the videos for transmission.

Based on the select signal, the MUX 7815 takes one input signal andoutputs the selected input signal to the encoder 7825. For instance, ifthe select signal indicates the MUX 7815 to take the input signal fromC1, the MUX 7815 selects that input signal and outputs it. The selectsignal can be provided in many ways such as through instructions fromthe video conference manager 1604. Through the MUX 7815, the encoder7825 alternately encodes images received from the MUX 7815 into a bitstream format and stores the encoded images in buffer 7830. Thecombining module 7835 combines (i.e. multiplexes) one or more bitstreams stored in the buffer 7830 and outputs a single bit stream.

The operation of this encoding approach will now be described in termsof three stages 7860, 7865, and 7870. In the first stage 7860, the MUX7815 is configured to receive and output the images 7805 captured bycamera C1 to the encoder 7825 for encoding. The encoder 7825 encodes thereceived images and generates a bit stream 7850, which is then stored inthe buffer 7830. The second stage 7865 is similar to the first stage7860 except the MUX 7815 is configured to receive and output the images7810 captured by camera C2 to the encoder 7825 for encoding. Again, theencoder encodes the received images and generates a bit stream 7855,which is stored in the buffer 7830. In the third stage 7870, thecombining module 7835 retrieves the bit streams 7850 and 7855 from thebuffer 7830 and combines them into one bit stream for transmission tothe remote device.

FIG. 79 illustrates another approach for encoding two videos from a dualcamera mobile device for transmission to a remote device during a videoconference. In this approach, a video frame (i.e. an image) from a firstcamera of the mobile device and another video frame from a second cameraof the mobile device are composited into one video frame before thecomposited video frame is encoded into a bit stream to be sent to theremote device. As shown in FIG. 79 , this approach includes a compositor7915, a buffer 7920, and an encoder 7925.

As shown, the compositor 7915 composites an image 7905 from the firstcamera and an image 7910 from the second camera to form a compositeimage 7955. Different embodiments composite the images 7905 and 7910differently. For instance, the compositor 7915 of some embodiments maycomposite the images by aligning the two images adjacent to one anotheras shown in FIG. 80 . Composite images 8030 and 8035 illustrate twoexample composite images using this technique. In the composite image8030, the image 7905 from the first camera is aligned on top of theimage 7910 from the second camera. Whereas, the composite image 8035shows the image 7905 aligned to the left of the image 7910.

In some other embodiments, the compositor 7915 may composite the twoimages 7905 and 7910 by superimposing the two images 7905 and 7910 ontop of a larger background image. A composite image 8040 of FIG. 80illustrates an example composite image using this technique. In thecomposite image 8040, the images 7905 and 7910 are aligned diagonallyand superimposed onto the blank image (i.e., the image 7905 is locatedon the top left corner and the image 7910 is located on the bottom rightcorner of the background image). In some embodiments, the camera sensorsmay be different sizes and thus capture images with different pixelresolutions. In such embodiments, the compositor 7915 may composite theimages 7905 and 7910 in a similar manner as illustrated by compositeimage 8045 of FIG. 80 . After compositing the two images, the compositor7915 store the composite images in the buffer 7920. The encoder 7925retrieves the composite images from the buffer 7920, encodes thecomposited images into a bit stream, and sends it to the remote deviceof the video conference.

The operations will now be described by reference to the compositor7915, the buffer 7920, and the encoder 7925 illustrated in FIG. 79 .First, a first camera sends an image 7905 as part of a sequence ofimages in a video to the compositor 7915. At the same time, a secondcamera sends another image 7910 as part of a sequence of images in avideo to the compositor 7915. The compositor 7915 then composites theimages 7905 and 7910 to form a composite image 7955 in ways that weredescribed above. Next, the compositor 7915 sends the composite image7955 to the buffer 7920. The buffer 7920 then stores the compositeimages before sending them to the encoder 7925. Finally, the encoder7925 encodes the composite images into a bit stream and sends it to theremote device of the video conference.

FIG. 81 illustrates yet another approach for encoding two videos from adual camera mobile device for transmission to a remote device during avideo conference. In this approach, the two videos from the device aredisplayed in a composite display, a screen shot of the composite displayis taken and encoded into a bit stream to send to the remote device. Asshown in FIG. 81 , this approach includes an encoder 8115. In someembodiments, the encoder 8115 encodes composite images and sends to theremote device.

The operations will now be described by reference to the encoder 8115illustrated in FIG. 81 . First, videos from the two cameras of a dualcamera mobile device are displayed on the device's screen in a compositedisplay. The composite display can present the videos in any manner. Forexample, the composite display in some embodiments can present the twovideos in a PIP display such as a PIP display 8105 illustrated in FIG.81 . In other embodiments, the composite display may present the twovideos in two side-by-side display areas or two diagonally aligneddisplay areas. Screenshots of the PIP display 8105, such as an image8110, are taken and sent to the encoder 8115. The encoder then encodesthe sequence of screenshots into a bit stream 8120 before sending it tothe remote device of the video conference. While several differentapproaches for encoding two videos are described above, other approachesare still possible.

6. Decoding

Some embodiments of the dual camera mobile device may receive bitstreams encoded by the approaches described above by reference to FIGS.78-81 . In such embodiments, the dual camera mobile device may receive(e.g., through the video conference control channel) informationindicating the approached used to encode the videos. FIG. 82 illustratesone approach of decoding bit streams of two videos received from anotherdevice through a communications network for display on the dual cameramobile device during a video conference. Specifically, this approach isused to decode bit streams encoded by the encoding approach described byreference to FIG. 78 , above.

As shown in FIG. 82 , this approach uses a separating module 8235,buffers 8230 and 8290, and a decoder module 8225. The separating module8235 breaks (i.e., demultiplexes) the bit streams into one or more bitstreams and stores the bit streams in the buffer 8230. The decoder 8225retrieves encoded bit streams, decodes them to generate videos, and thenstores the videos in the buffer 8290.

The operation of this approach will now be described by reference to theseparating module 8235, the buffers 8230 and 8290, and the decodermodule 8225 illustrated in FIG. 82 . First, the dual camera mobiledevice receives the bit stream 7845 (e.g., at the networking manager1614) through the communications network from the other device in thevideo conference. The separating module 8235 breaks the received bitstream into two bit streams 8255 and 8260 since the received bit streamis a multiplexed bit stream of two bit streams. Each encoded bit streamrepresents the video data captured from one out of the two cameras ofthe device. Then, the separating module 8235 stores the bit streams 8255and 8260 in the buffer 8230.

After, the decoder 8225 retrieves a bit stream 8250, which is one of thetwo bit streams 8255 and 8260, from the buffer 8230, the decoder 8225decodes the bit stream 8250 to generate video 8280, and stores the video8280 in the buffer 8290. The decoder 8225 also decodes the other of thebit streams 8255 and 8260 and stores the generated video in the buffer8290. Now, both videos can be retrieved from the buffer 8290 and storedor displayed on the dual camera mobile device.

FIG. 83 illustrates an approach to decoding bit streams that are encodedby the approach described by reference to FIG. 79 . As shown in FIG. 83, this approach includes a decoder 8325, a buffer 8320, and adecompositor 8315.

In some embodiments, the decoder 8325 receives a bit stream encoded bythe approach illustrated in FIG. 79 and decodes a bit stream into one ormore composite images, which are then stored in the buffer 8320. Thedecompositor 8315 extracts the two images from each composite image. Inorder to extract the two images from the composite images, thedecompositor 8315 also receives information indicating the location ofeach image within the composite image (e.g., information receivedthrough the video conference communication control channel from thedevice in the video conference that composited and encoded the images).

The operation of this approach will now be described by reference to thedecoder 8325, the buffer 8320, and the decompositor 8315 illustrated inFIG. 83 . First, the decoder 8325 receives a bit stream of video, suchas the one created by the approach described by reference to FIG. 79 ,from another mobile device of a video conference. The decoder 8325decodes the bit stream into one or more composite images, which includesthe composite image 7955 and stores them to the buffer 8320. The buffer8320 then stores the composite images before sending them to thedecompositor 8315. When the decompositor receives the composite image7955 from the buffer 8320, it breaks down the composite image 7955 intothe two images 7905 and 7910, which are identical to the images 7905 and7910 in FIG. 79 .

When a bit stream is received from a system such as the one described inFIG. 81 , a decoder such as the decoder 8325 in FIG. 83 decodes the bitstream into a sequence of screenshots. The sequence of screenshots isdisplayed on the device's screen as a video without further processing.

VI. Multiple Sources

As described above, videos can be captured by both cameras of the dualcamera mobile device and transmitted to another device in a videoconference. Rather than transmitting video captured from both cameras ofthe dual camera mobile device, some embodiments may transmit differentmedia content or any content displayed on the dual camera mobile devicealong with videos captured from a camera of the dual camera mobiledevice. In other words, these embodiments can transmit content from anumber of sources along with video captured by cameras of the dualcamera mobile device.

FIG. 84 conceptually illustrates another software architecture for avideo conferencing and processing module of a dual camera mobile deviceof some embodiments. The video conferencing and processing module ofFIG. 84 is the similar to the video conferencing and processing module1600 of FIG. 16 except the video conferencing and processing moduleincludes a display driver 8485 and a storage 8475, and the mediaexchange module 1620 includes a media source module 8470 and a screencapture module 8480.

The media source module 8470 of some embodiments routes media contentbetween the video conference module 8402 and the storage 8475. Examplesof media content include videos, images, documents, and music. Otherembodiments store other types of media content in the storage 8475. Thestorage 8475 of some embodiments is internal storage (e.g., RAM) whilethe storage 8475 of other embodiments is external storage (e.g., acompact flash (CF) card, a secure digital (SD) card, etc.).

In some embodiments, the screen capture module 8480 routes images ofcontent displayed on the display of the dual camera mobile devicethrough the display driver 8485. In some embodiments, the display driver8485 is responsible for capturing content on the display and convertingthe content into an image. Different embodiments capture differentcontent displayed on the display. For example, some embodiments captureall the content displayed on the display. Other embodiments capture aparticular display area of the display (e.g., a display area of acurrent active window, a display area of a PIP display, etc.).

Some example operations of the video conferencing and processing modulewill now be described by reference to FIG. 84 . To transmit mediacontent along with video captured from a camera of the dual cameramobile device, the video conference module 8402 of some embodimentsperforms the same operations as the video conference module 1602described above in FIG. 16 to except instead of retrieving images fromthe CIPU 1650, the video conference manager 1604 retrieves media contentfrom the storage 8475 through the media source module 8470. To transmitimages of content displayed on the display of the dual camera mobiledevice, some embodiments of the video conference manager 1604 retrieveimages of content displayed on the display of the dual camera mobiledevice through the display driver 8485. Some embodiments perform similarprocessing to the media content or images of content displayed on thedisplay (e.g., perspective correction, resizing, etc.) as performed onimages retrieved from the CIPU 1650 while other embodiments do notperform any processing at all.

The discussion above describes several of examples of transmittingcontent from various sources along with video captured by a camera ofthe dual camera mobile device. However, other embodiments can transmitother different types of content. For example, in a video conferenceinvolving multiple participants, some embodiments transmit videoreceived from one device on the video conference and video captured by acamera of the dual camera mobile device to another device. As such, anynumber of different types of content from any number of sources can betransmitted along with video captured by a camera of the dual cameramobile device.

VII. Multi-Participant Video Conference

The above sections that are related to video conferencing describe videoconferences with two participants. However, multi-participant videoconferencing (i.e., three or more participants) with the mobile devicesof some embodiments is also possible. In some embodiments, all theparticipants in a multi-participant video conference can view and hearone another. Other embodiments provide a multi-participant broadcastvideo conference in which one participant (e.g., the broadcaster) canview and hear all the other participants and all the other participantscan view and hear the broadcaster, but the other participants cannotview or hear each other (unless authorized by the broadcaster, forexample).

A. User Interface for Multi-Participant Video Conference

During a multi-participant video conference, some embodiments provide avariety of different UIs for displaying the participants of the videoconference and selecting particular participant(s) to view. For example,some embodiments of the mobile device provide a UI that simultaneouslydisplays all the participants of a multi-participant video conferenceand allows a user of the mobile device to select one of the participantsto view (e.g., by enlarging the selected participant's image). FIG. 85illustrates an example of such UI.

This figure illustrates a sequence of operations for simultaneousdisplaying of all the participants of the multi-participant videoconference and selecting of one of the participants to view in a UI 8530of a mobile device 8500 by reference to five different stages 8505,8510, 8515, 8520, and 8525 of the UI 8530. The first stage 8505illustrates the UI 8530 after a multi-participant video conference amongthree other users of other devices has been established. As shown, theUI 8530 includes a composite display 8535 and a display area 1155. Thecomposite display 8535 includes four display areas 8565, 8570, 8575, and8580 that display images captured by cameras of the participants of themulti-participant video conference. In this example, the display area8565 shows a user of the mobile device 8500 (i.e., the display area 8565displays images captured by the front camera of the mobile device 8500).The display area 1155 is the same as the display area 1155 previouslydescribed above in FIG. 12 .

The second stage 8510 shows the user of the mobile device 8500 startinga participant selection operation by selecting one of the display areasof the composite display area 8530. In particular, the second stage 8510shows the user selecting the display area 8570 (e.g., by tapping afinger 8550 on the display area 8570).

The third stage 8515 of the UI 8530 illustrates a composite display 8555after the participant selection operation is completed. Some embodimentsprovide an animation (not shown) to display a transition between thesecond stage 8510 and the third stage 8515. The composite display 8555includes a PIP display 8560 that shows the display area of theparticipant selected in the second stage 8510 (i.e., display area 8570)as the background display area and the display area 8565 of the user asthe inset display area of the PIP display 8560. In this example, the PIPdisplay 8560 shows the image of the selected display area 8570horizontally stretched to fit a landscape orientation. In someembodiments, the image is not stretched and the image of the selecteddisplay area maintains its portrait orientation (i.e., the extra spaceon each side of the background display area is filled with black bars asshown in FIG. 36 ). Furthermore, the composite display 8555 alsoincludes a composite display 8585 that shows scaled down images of thetwo unselected display areas 8575 and 8580.

The fourth stage 8520 shows the user of the mobile device 8500 startinga participant de-selection operation by selecting the PIP display 8560(e.g., by tapping a finger 8550 on the PIP display 8560). The fifthstage 8525 illustrates the composite display 8535 after the completionof the participant de-selection operation.

FIG. 85 shows an example sequence of operations for simultaneouslydisplaying all the participants of a multi-participant video conference,performing a participant selection operation, and performing aparticipant de-selection operation. Other sequences of operations arepossible. For instance, after the third stage 8515, instead of startingthe participant de-selection operation, the user can select one of theunselected display areas displayed in the composite display 8585 to swapthe newly selected display area in the display area 8585 with thebackground display area (i.e., the previously selected display area) ofthe PIP display 8560. As such, the user can swap display areas in thedisplay area 8585 with the background display area of the PIP display8560 at any time and any number of times during the multi-participantvideo conference. Also at any time during the multi-participant videoconference, the user can perform the participant de-selection operationto return to the composite display 8535. Moreover, different embodimentsallow the user to select a particular participant in different ways suchas by toggling a switch on the mobile device 8500, by giving voicecommands. etc.

Some embodiments provide techniques for automatically selectingparticipants based on speech detection, for example. In suchembodiments, when one of the participants speaks, the display area ofthat participant is automatically selected as the background displayarea of the PIP display 8560. When a different participant speaks, thedisplay area of that participant is automatically selected as thebackground display area of the PIP display 8560. In some embodiments,when none of the participants of the multi-participant video conferenceis speaking, the display displays the composite display 8535 after adefined amount of silence (e.g., 3 seconds). In some embodiments, whenthe user of the mobile device 8500 speaks, nothing happens on the UI8530 of the mobile device 8500.

FIG. 86 illustrates another example sequence of operations forsimultaneous displaying of all the participants of the multi-participantvideo conference and selecting one of the participants to view. FIG. 86illustrates this operation in a UI 8645 of the mobile device 8500 byreference to seven different stages 8505, 8605, 8610, 8615, 8620, 8625and 8630 of the UI 8645. The first stage 8505 is the same as the firststage 8505 illustrated in FIG. 85 as it shows the UI 8645 after amulti-participant video conference among three other users of otherdevices has been established.

The second stage 8605 illustrates the user of the mobile device 8500starting a participant selection operation by selecting the display area8570 (e.g., by placing two fingers on the display area 8570). The thirdstage 8610 shows a transitional stage of the participant selectionoperation. In this stage, the user is dragging the two fingers away fromeach other while causing the display area 8570 to become larger and fillup the display area of what used to be the composite display 8535. Thisexample shows the display area 8570 being selected, but any of the otherdisplay areas 8565, 8575, and 8580 can be selected. In some embodiments,the user of the mobile device 8500 is not allowed to select the displayarea of the user (i.e., display area 8565 in this example).

The fourth stage 8615 of the UI 8645 shows a PIP display 8635 of the UI8645 after the participant selection operation is completed. Someembodiments require the user to continue dragging the fingers away fromeach other until the display area 8570 fills the background display area8640 of the PIP display 8635 while other embodiments only require theuser's drag operation to be larger than a particular threshold amount(e.g., longer than a particular distance or longer than a particularamount of time) before the user removes the fingers. When the user'sdrag operation meets or exceeds the particular threshold amount, the UI8645 continues the enlarging of the display area 8570 until it fills thebackground display area 8640 of the PIP display 8635. Otherwise, theparticipant selection operation is not complete and the UI 8645 revertsback to the composite display 8535. As shown, the selected display area(i.e., display area 8570) is the background display area 8640 of the PIPdisplay 8635 and the display area 8565 of the user is the inset displayarea of the PIP display 8635. Some embodiments provide an animation (notshown) to display a transition between the third stage 8610 and thefourth stage 8615.

The fifth stage 8620 illustrates the user of the mobile device 8500starting a participant de-selection operation by selecting thebackground display area 8640 of the PIP display 8635 (e.g., by placingtwo fingers on the background display area 8640). The sixth stage 8625shows a transitional stage of the participant de-selection operation.The stage illustrates the user dragging the fingers toward each other toshrink the display area of what used to be the background display area8640 of the PIP display 8635. Similar to the operation described in thethird stage 8610, some embodiments require the user's drag operation tobe larger than a particular threshold amount (e.g., longer than aparticular distance or longer than a particular amount of time) beforethe user removes the fingers. Else, the participant de-selectionoperation is not complete and the UI 8645 reverts back to the PIPdisplay 8635. The seventh stage 8630 of the UI 8645 shows the compositedisplay 8535 after the completion of the participant de-selectionoperation.

FIG. 86 illustrates another example sequence of operations forsimultaneously displaying all the participants of a multi-participantvideo conference, performing a participant selection operation, andperforming a participant de-selection operation. However, someembodiments allow the user of the mobile device 8500 to repeatedlyperform a participant selection operation and participant dc-selectionoperation. FIG. 87 illustrates one such embodiment.

Specifically, FIG. 87 illustrates an example sequence of performing aparticipant selection operation and participant de-selection operationmultiple times in a UI 8730 by reference to seven different stage 8505,8705, 8615, 8710, 8715, 8720, and 8725 of the UI 8730. The first stage8505 is the same as the first stage 8505 of FIGS. 85 and 86 , mentionedabove. The second stage 8705 is similar to the second stage 8605 of FIG.86 except the user selects the display area 8570 by tapping the displayarea 8570 once (instead of placing two fingers on the display area8570). The third stage 8615 is the same as the fourth stage 8615 of FIG.86 as it shows the PIP display 8635 after the completion of theparticipant selection operation. The fourth stage 8710 is similar to thefifth stage 8620 of FIG. 86 except the user selects the backgrounddisplay area 8640 of the PIP display 8645 by tapping the backgrounddisplay area 8640 once (instead of placing two fingers on the backgrounddisplay area 8640).

The fifth stage 8715 is the same as the seventh stage 8630 of FIG. 86since it shows the composite display 8535 after the participantde-selection operation is completed. The sixth stage 8720 shows similarto the second stage 8510 except the participant selection operation isperformed on the display area 8575. Similarly, the seventh stage 8725 issimilar to the third stage 8705 as it shows the selected display area(i.e., display area 8575) as the background display area 8640 of the PIPdisplay 8635. Although FIG. 87 only shows a few participant selectionand participant de-selection operations, any number of such operationscan be performed during the multi-participant video conference.

Moreover, some embodiments provide UIs that can display differentlynumbers of participants during the video conference. For instance, theUI of some embodiments displays only some of the participants of themulti-participant video conference when the mobile device is held in anupright position (i.e., a portrait orientation) and displays additionalparticipants when the mobile device is held in a sideways position(i.e., a landscape orientation). Other embodiments display all theparticipants when the mobile device is held in the sideways position. Inaddition, some embodiments provide an animation to indicate thetransition between different positions and/or orientations of the mobiledevice that are similar to those illustrated in FIGS. 34, 35, 36 and 37. Other different animations are also possible.

As another example of a UI that displays different numbers ofparticipants during the video conference, some embodiments allow theuser of the mobile device to select multiple participants tosimultaneously view during the video conference. Referring to the firststage 8505 of FIG. 85 for purposes of explanation, some of theseembodiments allow the user of the mobile device 8500 to select two ormore of the display areas 8565, 8570, 8575, and 8580 (e.g., by tappingthe corresponding display areas in the composite display 8535). Theselected display areas can then be displayed in various manners such asa composite display, a PIP display, any of the display arrangementsillustrated in FIG. 65 , among other types of multi-participantsdisplays arrangements. Furthermore, although an example of someembodiments is described, one of ordinary skill will realize thatdifferent embodiments can select and display multiple participants of amulti-participant video conference any number of different ways.

B. User Interface for Multi-Participant Broadcast Video Conference

As noted above, a multi-participant broadcast video conference onlyallows one participant to hear and view all of the other participantswhile the other participants cannot hear or view each other. Tofacilitate multi-participant broadcast video conferences, someembodiments provide numerous different UIs for displaying thebroadcaster and the other participants of a multi-participant broadcastvideo conference. For example, some embodiments provide astudent-teacher-like UI layout similar to the layout of the third stage8515 illustrated in FIG. 85 . As such, the student-teacher UI layout ofsome embodiments will now be described by reference to this stage.

In these embodiments, only the broadcaster is displayed in the entiredisplay area of the PIP display 8560 (i.e., an inset display area is notdisplayed). The other participants of the multi-participant broadcastvideo conference are displayed below the PIP display 8560 similar to thedisplay areas displayed in composite display 8585. In some embodiments,a defined number of the other participants are displayed in thecomposite display 8585 when the mobile device is in a portrait modewhile additional or all participants can be displayed in the compositedisplay 8585 when the mobile device is in a landscape mode as similarlydescribed above. In addition, other embodiments provide different UIsfor displaying the broadcaster and the other participants of amulti-participant broadcast video conference.

C. Controlling Audio for Multi-Participant Video Conference

Further, the mobile device of some embodiments provides differenttechniques for controlling audio of participants of a multi-participantvideo conference. For example, some embodiments of the mobile deviceallow the user of the mobile device to control the audio of eachparticipant in the multi-participant video conference through a singleset of volume controls (e.g., a volume slider) displayed on the UI ofsuch embodiments. In other embodiments, the mobile device allows a userof the mobile device to separately control the volume of the audio ofeach participant in the multi-participant video conference through a setof volume controls such as a volume slider that is displayed in thedisplay area of each participant. Some embodiments only provide a mutebutton instead of a set of volume controls. As such, in some suchembodiments, the user of the mobile device can only mute or un-mute allthe participants of the multi-participant video conference while inother such embodiments the user of the mobile device can separately muteor un-mute each participant of the multi-participant video conference.In addition, other techniques for controlling the audio of participantsof the multi-participant video conference are possible such as bytoggling a switch on the mobile device, by giving voice commands, etc.

VIII. Electronic System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

Some embodiments are implemented as software processes that include oneor more application programming interfaces (APIs) in an environment withcalling program code interacting with other program code being calledthrough the one or more interfaces. Various function calls, messages orother types of invocations, which further may include various kinds ofparameters, can be transferred via the APIs between the calling programand the code being called. In addition, an API may provide the callingprogram code the ability to use data types or classes defined in the APIand implemented in the called program code.

At least certain embodiments include an environment with a callingsoftware component interacting with a called software component throughan API. A method for operating through an API in this environmentincludes transferring one or more function calls, messages, other typesof invocations or parameters via the API.

One or more Application Programming Interfaces (APIs) may be used insome embodiments. For example, some embodiments of the media exchangemodule 310 (or 910) provide a set of APIs to other software componentsfor accessing various video processing and encoding functionalitiesdescribed in FIGS. 3 and 9 such as the functionalities of the TNR module1900 described in FIG. 19 .

An API is an interface implemented by a program code component orhardware component (hereinafter “API-implementing component”) thatallows a different program code component or hardware component(hereinafter “API-calling component”) to access and use one or morefunctions, methods, procedures, data structures, classes, and/or otherservices provided by the API-implementing component. An API can defineone or more parameters that are passed between the API-calling componentand the API-implementing component.

An API allows a developer of an API-calling component (which may be athird party developer) to leverage specified features provided by anAPI-implementing component. There may be one API-calling component orthere may be more than one such component. An API can be a source codeinterface that a computer system or program library provides in order tosupport requests for services from an application. An operating system(OS) can have multiple APIs to allow applications running on the OS tocall one or more of those APIs, and a service (such as a programlibrary) can have multiple APIs to allow an application that uses theservice to call one or more of those APIs. An API can be specified interms of a programming language that can be interpreted or compiled whenan application is built.

In some embodiments the API-implementing component may provide more thanone API, each providing a different view of or with different aspectsthat access different aspects of the functionality implemented by theAPI-implementing component. For example, one API of an API-implementingcomponent can provide a first set of functions and can be exposed tothird party developers, and another API of the API-implementingcomponent can be hidden (not exposed) and provide a subset of the firstset of functions and also provide another set of functions, such astesting or debugging functions which are not in the first set offunctions. In other embodiments the API-implementing component mayitself call one or more other components via an underlying API and thusbe both an API-calling component and an API-implementing component.

An API defines the language and parameters that API-calling componentsuse when accessing and using specified features of the API-implementingcomponent. For example, an API-calling component accesses the specifiedfeatures of the API-implementing component through one or more API callsor invocations (embodied for example by function or method calls)exposed by the API and passes data and control information usingparameters via the API calls or invocations. The API-implementingcomponent may return a value through the API in response to an API callfrom an API-calling component. While the API defines the syntax andresult of an API call (e.g., how to invoke the API call and what the APIcall does), the API may not reveal how the API call accomplishes thefunction specified by the API call. Various API calls are transferredvia the one or more application programming interfaces between thecalling (API-calling component) and an API-implementing component.Transferring the API calls may include issuing, initiating, invoking,calling, receiving, returning, or responding to the function calls ormessages; in other words, transferring can describe actions by either ofthe API-calling component or the API-implementing component. Thefunction calls or other invocations of the API may send or receive oneor more parameters through a parameter list or other structure. Aparameter can be a constant, key, data structure, object, object class,variable, data type, pointer, array, list or a pointer to a function ormethod or another way to reference a data or other item to be passed viathe API.

Furthermore, data types or classes may be provided by the API andimplemented by the API-implementing component. Thus, the API-callingcomponent may declare variables, use pointers to, use or instantiateconstant values of such types or classes by using definitions providedin the API.

Generally, an API can be used to access a service or data provided bythe API-implementing component or to initiate performance of anoperation or computation provided by the API-implementing component. Byway of example, the API-implementing component and the API-callingcomponent may each be any one of an operating system, a library, adevice driver, an API, an application program, or other module (itshould be understood that the API-implementing component and theAPI-calling component may be the same or different type of module fromeach other). API-implementing components may in some cases be embodiedat least in part in firmware, microcode, or other hardware logic. Insome embodiments, an API may allow a client program to use the servicesprovided by a Software Development Kit (SDK) library. In otherembodiments an application or other client program may use an APIprovided by an Application Framework. In these embodiments theapplication or client program may incorporate calls to functions ormethods provided by the SDK and provided by the API or use data types orobjects defined in the SDK and provided by the API. An ApplicationFramework may in these embodiments provide a main event loop for aprogram that responds to various events defined by the Framework. TheAPI allows the application to specify the events and the responses tothe events using the Application Framework. In some implementations, anAPI call can report to an application the capabilities or state of ahardware device, including those related to aspects such as inputcapabilities and state, output capabilities and state, processingcapability, power state, storage capacity and state, communicationscapability, etc., and the API may be implemented in part by firmware,microcode, or other low level logic that executes in part on thehardware component.

The API-calling component may be a local component (i.e., on the samedata processing system as the API-implementing component) or a remotecomponent (i.e., on a different data processing system from theAPI-implementing component) that communicates with the API-implementingcomponent through the API over a network. It should be understood thatan API-implementing component may also act as an API-calling component(i.e., it may make API calls to an API exposed by a differentAPI-implementing component) and an API-calling component may also act asan API-implementing component by implementing an API that is exposed toa different API-calling component.

The API may allow multiple API-calling components written in differentprogramming languages to communicate with the API-implementing component(thus the API may include features for translating calls and returnsbetween the API-implementing component and the API-calling component);however the API may be implemented in terms of a specific programminglanguage. An API-calling component can, in one embodiment, call APIsfrom different providers such as a set of APIs from an OS provider andanother set of APIs from a plug-in provider and another set of APIs fromanother provider (e.g. the provider of a software library) or creator ofthe another set of APIs.

FIG. 88 is a block diagram illustrating an exemplary API architecture,which may be used in some embodiments of the invention. As shown in FIG.88 , the API architecture 8800 includes the API-implementing component8810 (e.g., an operating system, a library, a device driver, an API, anapplication program, software or other module) that implements the API8820. The API 8820 specifies one or more functions, methods, classes,objects, protocols, data structures, formats and/or other features ofthe API-implementing component that may be used by the API-callingcomponent 8830. The API 8820 can specify at least one calling conventionthat specifies how a function in the API-implementing component 8810receives parameters from the API-calling component 8830 and how thefunction returns a result to the API-calling component. The API-callingcomponent 8830 (e.g., an operating system, a library, a device driver,an API, an application program, software or other module), makes APIcalls through the API 8820 to access and use the features of theAPI-implementing component 8810 that are specified by the API 8820. TheAPI-implementing component 8810 may return a value through the API 8820to the API-calling component 8830 in response to an API call.

It will be appreciated that the API-implementing component 8810 mayinclude additional functions, methods, classes, data structures, and/orother features that are not specified through the API 8820 and are notavailable to the API-calling component 8830. It should be understoodthat the API-calling component 8830 may be on the same system as theAPI-implementing component 8810 or may be located remotely and accessesthe API-implementing component 8810 using the API 8820 over a network.While FIG. 88 illustrates a single API-calling component 8830interacting with the API 8820, it should be understood that otherAPI-calling components, which may be written in different languages (orthe same language) than the API-calling component 8830, may use the API8820.

The API-implementing component 8810, the API 8820, and the API-callingcomponent 8830 may be stored in a machine-readable medium, whichincludes any mechanism for storing information in a form readable by amachine (e.g., a computer or other data processing system). For example,a machine-readable medium includes magnetic disks, optical disks, randomaccess memory; read only memory, flash memory devices, etc.

FIG. 89 is an example of a dual camera mobile computing devicearchitecture 8900. The implementation of a mobile computing device caninclude one or more processing units 8905, memory interface 8910 and aperipherals interface 8915. Each of these components that make up thecomputing device architecture can be separate components or integratedin one or more integrated circuits. These various components can also becoupled together by one or more communication buses or signal lines.

The peripherals interface 8915 can be coupled to various sensors andsubsystems, including a camera subsystem 8920, a wireless communicationsubsystem(s) 8925, audio subsystem 8930, I/O subsystem 8935, etc. Theperipherals interface 8915 enables communication between processors andperipherals. Peripherals such as an orientation sensor 8945 or anacceleration sensor 8950 can be coupled to the peripherals interface8915 to facilitate the orientation and acceleration functions.

The camera subsystem 8920 can be coupled to one or more optical sensors8940, e.g., a charged coupled device (CCD) optical sensor, acomplementary metal-oxide-semiconductor (CMOS) optical sensor. Thecamera subsystem 8920 coupled with the sensors may facilitate camerafunctions, such as image and/or video data capturing. Wirelesscommunication subsystems 8925 may serve to facilitate communicationfunctions. Wireless communication subsystems 8925 may include radiofrequency receivers and transmitters, and optical receivers andtransmitters. They may be implemented to operate over one or morecommunication networks such as a GSM network, a Wi-Fi network, Bluetoothnetwork, etc. The audio subsystems 8930 is coupled to a speaker and amicrophone to facilitate voice-enabled functions, such as voicerecognition, digital recording, etc.

I/O subsystem 8935 involves the transfer between input/output peripheraldevices, such as a display, a touch screen, etc., and the data bus ofthe CPU through the Peripherals Interface. 1/O subsystem 8935 caninclude a touch-screen controller 8955 and other input controllers 8960to facilitate these functions. Touch-screen controller 8955 can becoupled to the touch screen 8965 and detect contact and movement on thescreen using any of multiple touch sensitivity technologies. Other inputcontrollers 8960 can be coupled to other input/control devices, such asone or more buttons.

Memory interface 8910 can be coupled to memory 8970, which can includehigh-speed random access memory and/or non-volatile memory such as flashmemory. Memory can store an operating system (OS) 8972. The OS 8972 caninclude instructions for handling basic system services and forperforming hardware dependent tasks.

Memory can also include communication instructions 8974 to facilitatecommunicating with one or more additional devices; graphical userinterface instructions 8976 to facilitate graphic user interfaceprocessing; image/video processing instructions 8978 to facilitateimage/video-related processing and functions; phone instructions 8980 tofacilitate phone-related processes and functions; media exchange andprocessing instructions 8982 to facilitate media communication andprocessing-related processes and functions; camera instructions 8984 tofacilitate camera-related processes and functions; and videoconferencing instructions 8986 to facilitate video conferencingprocesses and functions. The above identified instructions need not beimplemented as separate software programs or modules. Various functionsof mobile computing device can be implemented in hardware and/or insoftware, including in one or more signal processing and/or applicationspecific integrated circuits.

The above-described embodiments may include touch I/O device 9001 thatcan receive touch input for interacting with computing system 9003, asshown in FIG. 90 , via wired or wireless communication channel 9002.Touch I/O device 9001 may be used to provide user input to computingsystem 9003 in lieu of or in combination with other input devices suchas a keyboard, mouse, etc. One or more touch I/O devices 9001 may beused for providing user input to computing system 9003. Touch I/O device9001 may be an integral part of computing system 9003 (e.g., touchscreen on a laptop) or may be separate from computing system 9003.

Touch I/O device 9001 may include a touch sensitive panel which iswholly or partially transparent, semitransparent, non-transparent,opaque or any combination thereof. Touch I/O device 9001 may be embodiedas a touch screen, touch pad, a touch screen functioning as a touch pad(e.g., a touch screen replacing the touchpad of a laptop), a touchscreen or touchpad combined or incorporated with any other input device(e.g., a touch screen or touchpad disposed on a keyboard) or anymulti-dimensional object having a touch sensitive surface for receivingtouch input.

In one example, touch I/O device 9001 embodied as a touch screen mayinclude a transparent and/or semitransparent touch sensitive panelpartially or wholly positioned over at least a portion of a display.According to this embodiment, touch I/O device 9001 functions to displaygraphical data transmitted from computing system 9003 (and/or anothersource) and also functions to receive user input. In other embodiments,touch I/O device 9001 may be embodied as an integrated touch screenwhere touch sensitive components/devices are integral with displaycomponents/devices. In still other embodiments a touch screen may beused as a supplemental or additional display screen for displayingsupplemental or the same graphical data as a primary display andreceiving touch input.

Touch I/O device 9001 may be configured to detect the location of one ormore touches or near touches on device 9001 based on capacitive,resistive, optical, acoustic, inductive, mechanical, chemicalmeasurements, or any phenomena that can be measured with respect to theoccurrences of the one or more touches or near touches in proximity todevice 9001. Software, hardware, firmware or any combination thereof maybe used to process the measurements of the detected touches to identifyand track one or more gestures. A gesture may correspond to stationaryor non-stationary, single or multiple, touches or near touches on touchI/O device 9001. A gesture may be performed by moving one or morefingers or other objects in a particular manner on touch I/O device 9001such as tapping, pressing, rocking, scrubbing, twisting, changingorientation, pressing with varying pressure and the like at essentiallythe same time, contiguously, or consecutively. A gesture may becharacterized by, but is not limited to a pinching, sliding, swiping,rotating, flexing, dragging, or tapping motion between or with any otherfinger or fingers. A single gesture may be performed with one or morehands, by one or more users, or any combination thereof.

Computing system 9003 may drive a display with graphical data to displaya graphical user interface (GUI). The GUI may be configured to receivetouch input via touch I/O device 9001. Embodied as a touch screen, touchI/O device 9001 may display the GUI. Alternatively, the GUI may bedisplayed on a display separate from touch I/O device 9001. The GUI mayinclude graphical elements displayed at particular locations within theinterface. Graphical elements may include but are not limited to avariety of displayed virtual input devices including virtual scrollwheels, a virtual keyboard, virtual knobs, virtual buttons, any virtualUI, and the like. A user may perform gestures at one or more particularlocations on touch I/O device 9001 which may be associated with thegraphical elements of the GUI. In other embodiments, the user mayperform gestures at one or more locations that are independent of thelocations of graphical elements of the GUI. Gestures performed on touchI/O device 9001 may directly or indirectly manipulate, control, modify,move, actuate, initiate or generally affect graphical elements such ascursors, icons, media files, lists, text, all or portions of images, orthe like within the GUI. For instance, in the case of a touch screen, auser may directly interact with a graphical element by performing agesture over the graphical element on the touch screen. Alternatively, atouch pad generally provides indirect interaction. Gestures may alsoaffect non-displayed GUI elements (e.g., causing user interfaces toappear) or may affect other actions within computing system 9003 (e.g.,affect a state or mode of a GUI, application, or operating system).Gestures may or may not be performed on touch I/O device 9001 inconjunction with a displayed cursor. For instance, in the case in whichgestures are performed on a touchpad, a cursor (or pointer) may bedisplayed on a display screen or touch screen and the cursor may becontrolled via touch input on the touchpad to interact with graphicalobjects on the display screen. In other embodiments in which gesturesare performed directly on a touch screen, a user may interact directlywith objects on the touch screen, with or without a cursor or pointerbeing displayed on the touch screen.

Feedback may be provided to the user via communication channel 9002 inresponse to or based on the touch or near touches on touch I/O device9001. Feedback may be transmitted optically, mechanically, electrically,olfactory, acoustically, or the like or any combination thereof and in avariable or non-variable manner.

These functions described above can be implemented in digital electroniccircuitry, in computer software, firmware or hardware. The techniquescan be implemented using one or more computer program products.Programmable processors and computers can be included in or packaged asmobile devices. The processes and logic flows may be performed by one ormore programmable processors and by one or more programmable logiccircuitry. General and special purpose computing devices and storagedevices can be interconnected through communication networks.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic or other technological devices. These terms exclude people orgroups of people. For the purposes of the specification, the termsdisplay or displaying means displaying on an electronic device. As usedin this specification and any claims of this application, the terms“computer readable medium” and “computer readable media” are entirelyrestricted to tangible, physical objects that store information in aform that is readable by a computer. These terms exclude any wirelesssignals, wired download signals, and any other ephemeral signals.

FIG. 91 conceptually illustrates an example communication system 9100used for connecting some participants of a video conference according tosome embodiments. As shown, the communication system 9100 includesseveral mobile devices 9115, several cellular base stations (or Node Bs)9110, several radio network controllers (RNCs) 9105, and a core network9125. Cellular base stations and RNCs are collectively referred to as aUniversal Mobile Telecommunications System (UMTS) Terrestrial RadioAccess Network (UTRAN) 9130. Each RNC 9105 is connected to one or morecellular base stations 9110 that, together, are referred to as a radioaccess network (RAN).

Each cellular base station 9110 covers a service region 9120. As shown,the mobile devices 9115 in each service region are wirelessly connectedto the serving cellular base station 9110 of the service region 9120through a Uu interface. The Uu interface uses a protocol stack that hastwo planes: a control plane and a user plane. The user plane supportscircuit-switched, packet-switched and broadcast data streams. Thecontrol plane carries the network's signaling messages.

Each cellular base station is connected to an RNC through an Iubinterface. Each RNC 9105 is connected to the core network 9125 by Iu-csand an Iu-ps interfaces. The Iu-cs interface is used for circuitswitched services (e.g., voice) while the Iu-ps interface is used forpacket switched services (e.g., data). The Iur interface is used forconnecting two RNCs together.

Accordingly, the communication system 9100 supports bothcircuit-switched services and packet-switched services. For example,circuit-switched services allow a telephone call to be conducted bytransmitting the telephone call data (e.g., voice) throughcircuit-switched equipment of the communication system 9100.Packet-switched services allow a video conference to be conducted byusing a transport protocol layer such as UDP or TCP over an internetlayer protocol like IP to transmit video conference data throughpacket-switched equipment of the communication system 9100. In someembodiments, the telephone call to video conference transition (e.g.,handoff) previously described in the Video Conference Setup section usesthe circuit-switched and packet-switched services supported by acommunication system like the communication system 9100. That is, insuch embodiments, the telephone call is conducted through thecircuit-switched equipment of the communication system 9100 and thevideo conference it conducted through the packet-switched equipment ofthe communication system 9100.

Although the example communication system in FIG. 91 illustrates a thirdgeneration (3G) technology UTRAN wireless mobile communication system,it should be noted that second generation (2G) communication systems,other 3G communication systems such as 3GPP2 Evolution-Data Optimized orEvolution-Data only (EV-DO) and 3rd generation partnership project 2(3GPP2) Code Division Multiple Access 1× (CDMA 1×), fourth generation(4G) communication systems, wireless local area network (WLAN), andWorldwide Interoperability for Microwave Access (WiMAX) communicationsystems can be used for connecting some of the participants of aconference in some embodiments. Examples of 2G systems include GlobalSystem for Mobile communications (GSM), General Packet Radio Service(GPRS), and Enhanced Data Rates for GSM Evolution (EDGE). A 2Gcommunication system architecture is similar to the architecture shownin FIG. 91 except the 2G communication system architecture uses basetransceiver stations (BTSs) instead of Node Bs 9110 and base stationcontrollers (BSC) instead of RNC 9105. In a 2G communication system, anA interface between the BSC and the core network is used for circuitswitched services and a Gb interface between the BSC and the corenetwork is used for packet switched services.

In some embodiments, the communication system 9100 is operated by aservice carrier who initially provisions a mobile device 9115 to allowthe mobile device 9115 to use the communication system 9100. Someembodiments provision a mobile device 9115 by configuring andregistering a subscriber identity module (SIM) card in the mobile device9115. In other embodiments, the mobile device 9115 is instead configuredand registered using the mobile device 9115's memory. Moreover,additional services can be provisioned (after a customer purchases themobile device 9115) such as data services like GPRS, multimediamessaging service (MMS), and instant messaging. Once provisioned, themobile device 9115 is activated and is thereby allowed to use thecommunication system 9100 by the service carrier.

The communication system 9100 is a private communication network in someembodiments. In such embodiments, the mobile devices 9115 cancommunicate (e.g., conduct voice calls, exchange data) among each other(e.g., mobile devices 9115 that are provisioned for the communicationsystem 9100). In other embodiments, the communication system 9100 is apublic communication network. Thus, the mobile devices 9115 cancommunicate with other devices outside of the communication system 9100in addition to the mobile devices 9115 provisioned for the communicationsystem 9100. Some of the other devices outside of the communicationsystem 9100 include phones, computers, and other devices that connect tothe communication system 9100 through other networks such as a publicswitched telephone network or another wireless communication network.

The Long-Term Evolution (LTE) specification is used to define 4Gcommunication systems. FIG. 92 conceptually illustrates an example of a4G communication system 9200 that is used for connecting someparticipants of a video conference in some embodiments. As shown, thecommunication system 9200 includes several mobile devices 9115, severalEvolved Node Bs (eNBs) 9205, a Mobility Management Entity (MME) 9215, aServing Gateway (S-GW) 9220, a Packet Data Network (PDN) Gateway 9225,and a Home Subscriber Server (HSS) 9235. In some embodiments, thecommunication system 9200 includes one or more MMEs 9215, one or moreS-GWs 9220, one or more PDN Gateways 9225, and one or more HSSs 9235.

The eNBs 9205 provide an air interface for the mobile devices 9115. Asshown, each eNB 9205 covers a service region 9210. The mobile devices9115 in each service region 9210 are wirelessly connected to the eNB9205 of the service region 9210 through a LTE-Uu interface. FIG. 92 alsoshows the eNBs 9205 connected to each other through an X2 interface. Inaddition, the eNBs 9205 are connected to the MME 9215 through an SI-MMEinterface and to the S-GW 9220 through an S1-U interface. The eNBs 9205are collectively referred to as an Evolved UTRAN (E-TRAN) 9230.

The eNBs 9205 provide functions such as radio resource management (e.g.,radio bearer control, connection mobility control, etc.), routing ofuser plane data towards the S-GW 9220, signal measurement andmeasurement reporting, MME selection at the time of mobile deviceattachment, etc. The MME 9215 functions include idle mode mobile devicetracking and paging, activation and deactivation of radio bearers,selection of the S-GW 9220 at the time of mobile device attachment,Non-Access Stratum (NAS) signaling termination, user authentication byinteracting with the HSS 9235, etc.

The S-GW 9220 functions includes (1) routing and forwarding user datapackets and (2) managing and storing mobile device contexts such asparameters of the IP bearer service and network internal routinginformation. The PDN Gateway 9225 functions include providingconnectivity from the mobile devices to external packet data networks(not shown) by being the point of exit and entry of traffic for themobile devices. A mobile station may have simultaneous connectivity withmore than one PDN Gateway for accessing multiple packet data networks.The PDN Gateway 9225 also acts as the anchor for mobility between 3GPPand non-3GPP technologies such as WiMAX and 3GPP2 (e.g., CDMA 1× andEV-DO).

As shown, MME 9215 is connected to S-GW 9220 through an S11 interfaceand to the HSS 9235 through an S6a interface. The S-GW 9220 and the PDNGateway 9220 are connected through an S8 interface. The MME 9215, S-GW9220, and PDN Gateway 9225 are collectively referred to as an EvolvedPacket Core (EPC). The EPC is the main component of a SystemArchitecture Evolution (SAE) architecture, which is the core networkarchitecture of 3GPP LTE wireless communication standard. The EPC is apure packet system. For example, the EPC does not have a voice mediagateway. Services, like voice and SMS, are packet-switched routed andare provided by application functions that make use of the EPC service.So using the telephone call to video conference transition previouslydescribed above as an example, both the telephone call and the videoconference are conducted through packet-switched equipment of thecommunication system 9200 in some embodiments. In some such embodiments,the packet-switched channel used for the telephone call is continued tobe used for the audio data of the video conference after the telephonecall terminates. However, in other such embodiments, a differentpacket-switched channel is created (e.g., when the video conference isestablished) and audio data is transmitted through the newly createdpacket-switched channel instead of the packet-switched channel of thetelephone call when the telephone call terminates.

Moreover, the amount of bandwidth provided by these differenttechnologies ranges from 44 kilobits per second (kbps) for GPRS to over10 megabits per second (Mbps) for LTE. Download rates of 100 Mbps andupload rates of 50 Mbps are predicted in the future for LTE.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figuresconceptually illustrate processes. The specific operations of theseprocesses may not be performed in the exact order shown and described.The specific operations may not be performed in one continuous series ofoperations, and different specific operations may be performed indifferent embodiments. Furthermore, the process could be implementedusing several sub-processes, or as part of a larger macro process.

Also, many embodiments were described above by reference to a videoconference between two dual camera mobile devices. However, one ofordinary skill in the art will realize that many of these embodimentsare used in cases involving a video conference between a dual cameramobile device and another device, such as a single camera mobile device,a computer, a phone with video conference capability, etc. Moreover,many of the embodiments described above can be used in single cameramobile devices and other computing devices with video conferencecapabilities. Thus, one of ordinary skill in the art would understandthat the invention is not limited by the foregoing illustrative details,but rather is to be defined by the appended claims.

1-20. (canceled)
 21. A first electronic device, comprising: a touch-sensitive display screen; one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: presenting, on the first electronic device, a composite display having a first video captured by the first electronic device and a second video captured by a second electronic device that is engaged in a video conference with the first electronic device, the composite display comprising an inset display showing the first video and a main display showing the second video, wherein the inset display is smaller than the main display and overlapping at least a portion of the main display; during the video conference, receiving, at the first electronic device, a selection of the inset display, wherein the selection of the inset display includes contact at a location on the touch-sensitive display screen that is associated with the inset display; detecting that the location of the contact on the touch-sensitive display screen has moved towards a direction while the contact has been maintained; and in response to detecting that the location of the contact on the touch-sensitive display screen has moved towards a direction while the contact has been maintained, modifying the composite display by moving the inset display in a same direction of movement as the contact even after the contact with the touch-sensitive display screen has been terminated, wherein the inset display continues to move until it reaches a respective location within the composite display, the respective location designated for placing one side of the inset display.
 22. The first electronic device of claim 21, wherein the respective location is a snap location.
 23. The first electronic device of claim 21, wherein the respective location is a corner of the composite display.
 24. The first electronic device of claim 21, wherein the respective location is a corner of the touch-sensitive display screen.
 25. The first electronic device of claim 21, wherein moving the inset display even after the contact with the touch-sensitive display screen has been terminated includes: moving the inset display after the contact has been terminated if the movement of the contact from the location exceeds a threshold amount; and moving the inset display back to its initial location after the contact has terminated if the movement of the contact from the location does not exceed the threshold amount.
 26. The first electronic device of claim 21, wherein moving the inset display includes moving the inset display from a first corner of the composite display to a second corner of the composite display.
 27. The first electronic device of claim 21, the one or more programs further including instructions for: detecting a rotation of the touch-sensitive display screen of the first electronic device; and in response to detecting the rotation of the touch-sensitive display screen of the first electronic device, rotating the composite display.
 28. The first electronic device of claim 21, wherein the selection of the inset display includes a selection of a part of the inset display and a movement of the part, and wherein modifying the composite display further comprises changing the size of the inset display based on the selection of the part of the inset display and the movement of the part.
 29. The first electronic device of claim 28, wherein the movement of the part comprises a movement of the part toward a center of the inset display, and the modification of the composite display comprises shrinking of a size of the inset display.
 30. The first electronic device of claim 28, wherein the movement of the part comprises a movement of the part in a direction away from a center of the inset display, and wherein the modification of the composite display comprises enlarging of a size of the inset display.
 31. The first electronic device of claim 21, wherein modifying the composite display further comprises swapping the inset display and the main display such that the inset display shows the second video and the main display shows the first video.
 32. The first electronic device of claim 21, wherein modifying the composite display comprises moving the first video with respect to the composite display.
 33. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a first electronic device that includes a touch-sensitive display screen, the one or more programs including instructions for: presenting, on the first electronic device, a composite display having a first video captured by the first electronic device and a second video captured by a second electronic device that is engaged in a video conference with the first electronic device, the composite display comprising an inset display showing the first video and a main display showing the second video, wherein the inset display is smaller than the main display and overlapping at least a portion of the main display; during the video conference, receiving, at the first electronic device, a selection of the inset display, wherein the selection of the inset display includes contact at a location on the touch-sensitive display screen that is associated with the inset display; detecting that the location of the contact on the touch-sensitive display screen has moved towards a direction while the contact has been maintained; and in response to detecting that the location of the contact on the touch-sensitive display screen has moved towards a direction while the contact has been maintained, modifying the composite display by moving the inset display in a same direction of movement as the contact even after the contact with the touch-sensitive display screen has been terminated, wherein the inset display continues to move until it reaches a respective location within the composite display, the respective location designated for placing one side of the inset display.
 34. A method, comprising: at a first electronic device that includes a touch-sensitive display screen: presenting, on the first electronic device, a composite display having a first video captured by the first electronic device and a second video captured by a second electronic device that is engaged in a video conference with the first electronic device, the composite display comprising an inset display showing the first video and a main display showing the second video, wherein the inset display is smaller than the main display and overlapping at least a portion of the main display; during the video conference, receiving, at the first electronic device, a selection of the inset display, wherein the selection of the inset display includes contact at a location on the touch-sensitive display screen that is associated with the inset display; detecting that the location of the contact on the touch-sensitive display screen has moved towards a direction while the contact has been maintained; and in response to detecting that the location of the contact on the touch-sensitive display screen has moved towards a direction while the contact has been maintained, modifying the composite display by moving the inset display in a same direction of movement as the contact even after the contact with the touch-sensitive display screen has been terminated, wherein the inset display continues to move until it reaches a respective location within the composite display, the respective location designated for placing one side of the inset display. 